Video World Models · 2026-11-06
FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
Haitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FewStepRL adapts reinforcement learning objectives for fine-tuning diffusion models when using few-step (average-velocity) samplers. The authors transform RL objectives to x-space, decompose gradients into self-regression plus reward augmentation, and adapt each objective’s anchors for few-step transport or tangents. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD, and ReFL in one modular codebase.
Why it matters
It addresses the mismatch between RL objectives built for instantaneous velocity (typical with many-step samplers) and few-step models that model average velocity over an interval. The paper provides a systematic adaptation framework and empirical guidance on when reward-gradient-based methods and group-advantage methods work best, including observations about inference-step scaling.
Method
- Transform RL objectives to x-space using a common policy prediction (clean end by default; explicitly marked for transition/flow-map objectives) so density/velocity/reward objectives become comparable.
- Decompose each objective gradient into a self-regression term toward a reference plus a reward-augmentation term; at the gradient level, differences reduce to choices of anchors and weights.
- Adapt anchors to few-step models by computing them from the few-step model at transport end or along the tangent, recovering instantaneous velocity either from the network itself or an auxiliary network; implement many few-step trainers in one modular codebase.
Limitation
Extra inference steps show no systematic aggregate benefit in the experiments, though selected outputs visibly improve with two to four times as many sampling steps as in training.
Abstract (from arXiv)
Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to $x$-space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.
Related papers
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
- Statistical Foundations of Reinforcement Learning: A Non-Asymptotic Perspective
- DPDP: Diffusion Policy-Steered Diffusion Planner for Offline Reinforcement Learning
- CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-05 · Indexed by Pub-AI: 2026-11-06 UTC
FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
Preprint posted: 2026-11-05 · Original record ↗
- arxiv
- 2611.08240
Explicit source versions (1)
- v1 · 2026-11-05
- Preprint available
Source date: 2026-11-05 · First detected: 2026-11-06 UTC · Historical baseline
Source evidence (3)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.08240v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to $x$-space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.
