Driving · 2026-10-23
DiffDreamer: Coupling World Models with Differentiable Simulation for Agile Quadrotor Control
Fanxing Li, Yuxiang Huang, Shengyang Wang, Fangyu Sun, Shuyu Wu, Tianbao Zhang, Dexin Zuo, Yan Yufei, Danping Zou, Wenxian Yu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DiffDreamer couples a learned world model with a differentiable quadrotor simulator for actor optimization. In one step, the world model propagates latent states for external, history-dependent dynamics while the simulator propagates the quadrotor’s explicit physical state. The actor objective is split into a critic-driven term for non-differentiable task rewards and a simulator-driven term for differentiable motion rewards, improving training convergence and final performance in aggressive tracking and dynamic-gate racing.
Why it matters
The authors address model bias and less accurate gradients that can arise when learned rollouts entangle external-motion dynamics with the robot’s high-frequency rigid-body dynamics. By structurally separating latent external dynamics from explicit differentiable physics and synchronizing both under the same action, DiffDreamer aims to improve optimization accuracy for agile quadrotor control.
Method
- Couples a learned RSSM world model (latent state capturing history-dependent external dynamics) with a differentiable simulation (propagates explicit quadrotor physical state) synchronized by the shared action sequence.
- Decomposes actor training into a critic-driven pathway for non-differentiable/discrete task rewards and a simulator-driven pathway optimized via backpropagation through differentiable motion rewards.
- Uses an odometry-conditioned joint state (implicit latent + explicit physical) and stop-gradient on latent inputs so motion-reward gradients flow through the physical rollout without backpropagating through the imagined latent input.
Limitation
DiffDreamer has been evaluated only on quadrotors. Extending its coupled latent/physical rollouts and reward decomposition to other embodiments requires suitable differentiable models and additional task-specific engineering. Contacts, discontinuities, actuator limits, and perception modules can complicate differentiation. Real-world validation relies on motion-capture feedback for both the vehicle and external objects. Fully onboard perception and robustness to perception noise remain untested.
Abstract (from arXiv)
Tasks such as quadrotor dynamic-gate racing require agents to anticipate external motion while maintaining precise high-frequency control. World-model-based reinforcement learning provides a natural solution by learning latent dynamics for future imagination, yet it often entangles environmental evolution and the robot's own dynamics in a single learned rollout, where model bias can affect policy optimization efficiency. We propose DiffDreamer, a differentiable reinforcement learning framework that tightly couples a learned world model with differentiable simulation for actor optimization. In one optimization step, the learned world model propagates latent states that capture external dynamics and history-dependent features, while the differentiable simulator propagates the robot's explicit physical state. We further decompose the actor objective into a critic-driven pathway for non-differentiable task rewards and a simulator-driven pathway for differentiable motion rewards. Experiments on aggressive tracking and dynamic-gate racing show that DiffDreamer improves training convergence over the model-free, model-based, and differentiable-RL baselines, and final performance over these baselines and classical planning and control methods. Additional experiments assess reward decomposition and the odometry-conditioned relative-state representation through ablations. We also provide proof-of-concept real-world flight experiments for validation.
Related papers
- DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
- CausalDreamer: Learning Predictive World Models with Latent Disentanglement
- PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Visual Subgoals and Policy Priors
- Mastering Atari with Discrete World Models
- SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC
DiffDreamer: Coupling World Models with Differentiable Simulation for Agile Quadrotor Control
Preprint posted: 2026-10-22 · Original record ↗
- arxiv
- 2610.31617
Explicit source versions (1)
- v1 · 2026-10-22
- Preprint available
Source date: 2026-10-22 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Tasks such as quadrotor dynamic-gate racing require agents to anticipate external motion while maintaining precise high-frequency control. World-model-based reinforcement learning provides a natural solution by learning latent dynamics for future imagination, yet it often entangles environmental evolution and the robot's own dynamics in a single learned rollout, where model bias can affect policy optimization efficiency. We propose DiffDreamer, a differentiable reinforcement learning framework that tightly couples a learned world model with differentiable simulation for actor optimization. In one optimization step, the learned world model propagates latent states that capture external dynamics and history-dependent features, while the differentiable simulator propagates the robot's explicit physical state. We further decompose the actor objective into a critic-driven pathway for non-differentiable task rewards and a simulator-driven pathway for differentiable motion rewards. Experiments on aggressive tracking and dynamic-gate racing show that DiffDreamer improves training convergence over the model-free, model-based, and differentiable-RL baselines, and final performance over these baselines and classical planning and control methods. Additional experiments assess reward decomposition and the odometry-conditioned relative-state representation through ablations. We also provide proof-of-concept real-world flight experiments for validation.