Driving · 2026-10-06
Reachability-Aware Diffusion Policy Optimization
Hikmet Simsir, Kutay Demiray, Ozgur S. Oguz
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RADPO (Reachability-Aware Diffusion Policy Optimization) is a model-free online diffusion-policy method for continuous-control safety. It learns a discounted first-hit reachability value to shape a reward critic, and uses an adaptive safety multiplier updated from realized episodic costs versus a budget. The diffusion actor is trained via value-weighted denoising regression on candidate actions, requiring no dynamics model, critic action gradients, or differentiation through the reverse diffusion sampler.
Why it matters
The authors target safety-aware online diffusion-policy optimization using predictive reachability without an explicit learned dynamics model. RADPO separates predictive “when cost first happens” shaping (reachability) from budget regulation (multiplier), and provides theoretical bounds relating reachability penalties to discounted cumulative cost, plus finite-candidate policy-improvement guarantees.
Method
- Learn an ensemble of bounded discounted first-hit reachability critics from replay; use the ensemble mean to shape the reward critic via r~_t = r_t - β λ V_reach(s_{t+1}).
- Train twin reward critics on shaped rewards (with bootstrap actions) and update the diffusion actor using value-weighted denoising regression on K candidate actions scored by the reward critic.
- Update an adaptive safety multiplier with a projected controller rule based on realized episodic costs in a recent window; reward shaping strength scales with observed budget violations.
Abstract (from arXiv)
Diffusion policies provide expressive action distributions for continuous-control reinforcement learning. However, safety-aware online diffusion policy optimization remains underexplored, particularly methods that use predictive reachability information without an explicit dynamics model. We propose Reachability-Aware Diffusion Policy Optimization (RADPO), a model-free method that combines predictive first-hit safety estimation with cumulative-cost budget feedback. RADPO learns a discounted first-hit reachability value that captures the discounted risk of a cost event, assigns larger weight to events that occur sooner, and uses this signal to shape the reward. A separate dual-like multiplier adjusts the shaping strength according to realized episodic costs relative to a prescribed budget. The diffusion actor improves through weighted denoising regression on candidate actions scored by the reward critic. Our approach requires neither a learned dynamics model, action gradients through the critics, nor differentiation through the reverse diffusion sampler. We establish theoretical properties of the reachability value and show that accumulated reachability penalty provides a conservative surrogate for future discounted cumulative cost. Across ten continuous-control safety tasks, RADPO achieves competitive reward-cost trade-offs, with substantial reductions in constraint violations on several tasks relative to the compared baselines. Our theoretical and empirical analysis supports that combining reachability with cumulative budget feedback is a viable approach to safety-aware diffusion policies.
Related papers
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
- Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
- Higher-Order Action Supervision Makes A Strong Policy Class
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC
Reachability-Aware Diffusion Policy Optimization
Preprint posted: 2026-10-05 · Original record ↗
- arxiv
- 2610.05969
Explicit source versions (1)
- v1 · 2026-10-05
- Preprint available
Source date: 2026-10-05 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Diffusion policies provide expressive action distributions for continuous-control reinforcement learning. However, safety-aware online diffusion policy optimization remains underexplored, particularly methods that use predictive reachability information without an explicit dynamics model. We propose Reachability-Aware Diffusion Policy Optimization (RADPO), a model-free method that combines predictive first-hit safety estimation with cumulative-cost budget feedback. RADPO learns a discounted first-hit reachability value that captures the discounted risk of a cost event, assigns larger weight to events that occur sooner, and uses this signal to shape the reward. A separate dual-like multiplier adjusts the shaping strength according to realized episodic costs relative to a prescribed budget. The diffusion actor improves through weighted denoising regression on candidate actions scored by the reward critic. Our approach requires neither a learned dynamics model, action gradients through the critics, nor differentiation through the reverse diffusion sampler. We establish theoretical properties of the reachability value and show that accumulated reachability penalty provides a conservative surrogate for future discounted cumulative cost. Across ten continuous-control safety tasks, RADPO achieves competitive reward-cost trade-offs, with substantial reductions in constraint violations on several tasks relative to the compared baselines. Our theoretical and empirical analysis supports that combining reachability with cumulative budget feedback is a viable approach to safety-aware diffusion policies.