Latent Dynamics · 2026-10-15
Decoupled Drift-Diffusion World Models for Consistent Future Frame Prediction
Anton Arapin, Aditya Singh, Yixin Wang, Jun Gao, Bernadette Bucher
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
D3WM predicts future image frames from action sequences by separating deterministic “drift” from stochastic “diffusion.” It trains a transformer to output a deterministic next-state mean plus a spatial heteroscedastic aleatoric uncertainty (variance) map. A variance-gated conditional diffusion model then refines uncertain regions while anchoring deterministic regions, yielding improved action-conditioned prediction on robotic manipulation and camera-motion/NVS.
Why it matters
The authors’ key claim is that the spatial location of stochastic scene changes is learnable. By predicting a structured uncertainty/variance map and using it to gate conditional diffusion, D3WM aims to preserve deterministic geometric structure (e.g., under disocclusions) while sampling plausible content only where deterministic prediction is uncertain, improving usefulness for downstream trajectory planning and evaluation.
Method
- Train a transformer with a Gaussian NLL objective to jointly regress the deterministic future latent mean μ_t and a spatial aleatoric variance map σ_t^2 from (z0, action sequence).
- Condition a ControlNet-based conditional diffusion model on (μ_t, σ_t^2) (and the reference image) to sample refinements, using σ_t^2 as a spatial gate at inference.
- Implement variance-gated inference via a VP-DDIM reverse process modification: at each reverse step, blend deterministic and diffusion latents using a per-pixel trust map derived from (1 − σ_t^2) and a diffusion-time anchoring schedule.
Limitation
The authors note that D3WM sometimes samples not the most precise stochastic outcomes in terms of end-effector shape and object colors and textures, but preserves the generally correct motion due to deterministic prediction grounding (BridgeData V2 discussion).
Abstract (from arXiv)
World models of visual information predict future image frames conditioned on given action sequences. This predicted scene evolution is driven by deterministic changes such as controlled motion on a robot platform, as well as stochastic changes such as rendering of manipulated objects or previously occluded regions. Our key insight is that the spatial location of these stochastic changes is itself learnable. We leverage this insight by jointly regressing both the deterministic next-state prediction and the spatial uncertainty map, then using the uncertainty map to guide a conditional diffusion process. We propose Decoupled Drift-Diffusion World Model (D3WM), which uses a transformer trained with a Gaussian negative-log-likelihood objective to produce both the deterministic next-state prediction and a spatial map of heteroscedastic aleatoric uncertainty. The learned uncertainty map then provides a structured spatial gate for a conditional diffusion model: deterministic regions are anchored to the predicted state, while uncertain regions, such as disocclusions or dynamic textures, are sampled by the diffusion process. We frame this decomposition under a variance-gated stochastic differential equation that separates deterministic drift from stochastic diffusion. D3WM shows improved results over competitive baselines in two distinct types of action spaces: robotic tabletop manipulation and camera motion.
Related papers
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
- TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
- Identifiable World Models from Pretrained Diffusion Representations
- Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation
- FutureWorlds: Learning Robotic World Models from Alternative Futures
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-15 UTC
Decoupled Drift-Diffusion World Models for Consistent Future Frame Prediction
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.19212
Explicit source versions (1)
- v1 · 2026-10-13
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models of visual information predict future image frames conditioned on given action sequences. This predicted scene evolution is driven by deterministic changes such as controlled motion on a robot platform, as well as stochastic changes such as rendering of manipulated objects or previously occluded regions. Our key insight is that the spatial location of these stochastic changes is itself learnable. We leverage this insight by jointly regressing both the deterministic next-state prediction and the spatial uncertainty map, then using the uncertainty map to guide a conditional diffusion process. We propose Decoupled Drift-Diffusion World Model (D3WM), which uses a transformer trained with a Gaussian negative-log-likelihood objective to produce both the deterministic next-state prediction and a spatial map of heteroscedastic aleatoric uncertainty. The learned uncertainty map then provides a structured spatial gate for a conditional diffusion model: deterministic regions are anchored to the predicted state, while uncertain regions, such as disocclusions or dynamic textures, are sampled by the diffusion process. We frame this decomposition under a variance-gated stochastic differential equation that separates deterministic drift from stochastic diffusion. D3WM shows improved results over competitive baselines in two distinct types of action spaces: robotic tabletop manipulation and camera motion.