DDPO for diffusion fine-tuning via PPO-style policy gradients with end-of-trajectory rewards
Denoising Diffusion Policy Optimization (DDPO) reframes diffusion denoising as a multi-step Markov decision process and fine-tunes the diffusion model using a PPO-style policy-gradient update, where a reward is applied at the end of the denoising trajectory. In an RLHF-like workflow for image generation, a reward model (e.g., an aesthetic predictor) scores generated samples, and those rewards drive DDPO to adjust the diffusion model toward the objective captured by the reward.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
The approach is designed to optimize over the whole denoising trajectory rather than only weighting a loss tied to the final sample. The source describes DDPO as avoiding certain approximations used by reward-weighted regression (RWR) by treating each denoising step in the optimization setup and using an end-of-trajectory reward within an MDP formulation.
What changed
Compared with reward-weighted regression (RWR), DDPO changes the optimization framing: instead of focusing only on the final denoised sample while ignoring intermediate denoising steps, DDPO treats the full denoising process as a sequence of decisions and applies reinforcement learning with PPO-style updates based on a reward received at the end of the trajectory.
Where it helps
- Fine-tune Stable Diffusion toward an aesthetic or preference-based reward
- RLHF-like alignment where a learned reward model scores generated images
- Experimenting with custom reward signals beyond a prompt objective function
- Preference-driven optimization of diffusion outputs using TRL’s DDPOTrainer
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The TRL DDPOTrainer mentioned in the source is reported to be limited to fine-tuning vanilla Stable Diffusion models; the source primarily reports experiments using LoRA and notes that non-LoRA runs require more careful hyperparameter selection.
Sources
- Finetune Stable Diffusion Models with DDPO via TRL ↗
Hugging Face · primary · Published 2023-09-29 · Checked 2026-11-10
