← AI Techniques Radar
Training & fine-tuning

DDPO for diffusion fine-tuning via PPO-style policy gradients with end-of-trajectory rewards

Denoising Diffusion Policy Optimization (DDPO) reframes diffusion denoising as a multi-step Markov decision process and fine-tunes the diffusion model using a PPO-style policy-gradient update, where a reward is applied at the end of the denoising trajectory. In an RLHF-like workflow for image generation, a reward model (e.g., an aesthetic predictor) scores generated samples, and those rewards drive DDPO to adjust the diffusion model toward the objective captured by the reward.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

The approach is designed to optimize over the whole denoising trajectory rather than only weighting a loss tied to the final sample. The source describes DDPO as avoiding certain approximations used by reward-weighted regression (RWR) by treating each denoising step in the optimization setup and using an end-of-trajectory reward within an MDP formulation.

What changed

Compared with reward-weighted regression (RWR), DDPO changes the optimization framing: instead of focusing only on the final denoised sample while ignoring intermediate denoising steps, DDPO treats the full denoising process as a sequence of decisions and applies reinforcement learning with PPO-style updates based on a reward received at the end of the trajectory.

Where it helps

  • Fine-tune Stable Diffusion toward an aesthetic or preference-based reward
  • RLHF-like alignment where a learned reward model scores generated images
  • Experimenting with custom reward signals beyond a prompt objective function
  • Preference-driven optimization of diffusion outputs using TRL’s DDPOTrainer

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The TRL DDPOTrainer mentioned in the source is reported to be limited to fine-tuning vanilla Stable Diffusion models; the source primarily reports experiments using LoRA and notes that non-LoRA runs require more careful hyperparameter selection.

Sources