Driving · 2026-10-14
CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
Shutong Ding, Zhaoyang Liu, Yimiao Zhou, Jingya Wang, Jiayuan Gu, Ye Shi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CFG-Policy is a plug-and-play diffusion/flow RL framework that separates exploration and exploitation into two generative policies and composes their score/velocity fields during sampling via classifier-free guidance. In online RL, the exploration branch is frozen while an exploitation branch learns from task rewards; in offline-to-online post-training, the pretrained exploitation branch is frozen while a trainable branch steers composed behavior. The authors report improved results with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin tasks.
Why it matters
The authors address diffusion RL’s exploration–exploitation trade-off by preserving either an exploration capability (online RL) or pretrained behavior/capabilities (offline-to-online post-training) while still optimizing the underlying RL objective. Their method can be added to compatible diffusion RL/flow RL backbones by replacing only the sampler via guided composition, aiming to improve performance while retaining diversity or pretrained functionality.
Method
- Unified framework: maintain two generative policies (exploration and exploitation) and combine their score/velocity fields at each sampling step using a classifier-free-guidance weight w.
- Online RL: freeze exploration policy πE, train exploitation policy πX using the original RL objectives; composed policy serves for behavior/data collection.
- RL post-training: freeze pretrained exploitation policy πX0 and train a steering policy πEθ; keep base RL objective unchanged while stopping gradients through the frozen branch.
Limitation
The preferred guidance weight w varies substantially across tasks. Although adaptive guidance reduces the need for manual selection, task-specific tuning can still find weights that outperform it.
Abstract (from arXiv)
Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at https://zhy-liu.github.io/cfg-policy-page.
Related papers
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
- Don't Throw Away the Tail: Action Upcycling for Policy Acceleration
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
- Reachability-Aware Diffusion Policy Optimization
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-10 · Indexed by Pub-AI: 2026-10-14 UTC
CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
Preprint posted: 2026-10-10 · Original record ↗
- arxiv
- 2610.14936
Explicit source versions (1)
- v1 · 2026-10-10
- Preprint available
Source date: 2026-10-10 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at https://zhy-liu.github.io/cfg-policy-page.