Video World Models · 2026-10-12
DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DiffusionART fine-tunes diffusion models with black-box rewards in an efficient and stable way. The authors diagnose instability in NFT-style forward-process RL as a hidden forward–reverse consistency force and argue EMA stabilizes via restoration but needs per-task tuning. DiffusionART aggregates each prompt’s rewards into a shared reward-improving direction for reverse states and uses a trust-region-masked two-update restoration, improving stability without EMA tuning.
Why it matters
The paper targets online diffusion RL that must handle output-level black-box rewards with credit assignment across many denoising states. The authors propose an approach that removes NFT’s consistency-force issue by construction and replaces EMA tuning with drift-adaptive restoration, aiming for scalable training across image and video generation without per-task tuning.
Method
- Diagnoses NFT-style forward-process training as containing a hidden forward–reverse consistency force; joint positive–negative training cancels it only partially.
- Introduces DiffusionART: aggregates each prompt’s endpoint rewards into a shared covariance-derived guidance direction and applies it to reverse-process states (removing the NFT consistency term by construction).
- Adds two-update drift-adaptive restoration (split rollout batch into two disjoint minibatches) plus a trust-region-masked second update to avoid excessive policy updates without EMA schedule tuning.
Limitation
The EMA trick used to stabilize NFT “must be tuned for each task,” and its schedule therefore requires task-specific tuning (Section 3.2).
Abstract (from arXiv)
Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.
Related papers
- CAST: Causal Advantage-Structured Training with Spatially Grounded Compositional Rewards for Diffusion Models
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
- Reachability-Aware Diffusion Policy Optimization
- VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-12 UTC
DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
Preprint posted: 2026-10-08 · Original record ↗
- arxiv
- 2610.12572
Explicit source versions (1)
- v1 · 2026-10-08
- Preprint available
Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.