Driving · 2026-11-04
Future Representation Distillation for Imagination-free World Action Models
Jaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DreamFree is an imagination-free World Action Model that repurposes a pretrained video diffusion model as a predictive visual encoder via future representation distillation. A visual encoder predicts a frozen diffusion model’s representations from the current observation only, enabling action prediction without generating future videos. The authors report higher success rates than state-of-the-art VLA and WAM baselines and real-time closed-loop control at 11Hz with a 14B video backbone.
Why it matters
The work targets two deployment issues of video-generating WAMs: costly iterative video denoising and sensitivity of action prediction to inaccurate future visual forecasts. By distilling predictive knowledge into features computed from the current observation, DreamFree aims to deliver predictive priors for control without future video generation, while maintaining strong manipulation performance and real-time latency.
Method
- Train a predictive visual encoder (initialized from a pretrained video diffusion backbone) to predict representations of a frozen video diffusion model using only the current observation and language instruction (future representation distillation).
- Separate training into (1) predictive representation learning with distillation + native generative objective, and (2) action learning with an action module (readout + action decoder) trained from robot demonstrations.
- During inference, run a single forward pass through the predictive visual encoder to extract action-relevant tokens; generate an action chunk without iterative video denoising (supports up to 11Hz for a 14B backbone).
Limitation
Scaling with embodied pretraining: the authors demonstrate a 14B video backbone trained on DROID without embodied pretraining can outperform existing VLA and WAM baselines, but RoboCasa results suggest additional video data could improve performance; they did not evaluate this extension on DROID due to computational constraints. They also note training complexity from separating backbone adaptation, predictive representation learning, and action learning, and propose joint training as future work.
Abstract (from arXiv)
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Related papers
- Self-Distilling Future Video Representations for World Action Models
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
- W2A: Distilling World Priors into Action Models for Real-Time Robotic Control
- SUAVE: Unified Video-Action Models via Masked Diffusion
- Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC
Future Representation Distillation for Imagination-free World Action Models
First announced by arXiv: 2026-11-04 · Original record ↗
- arxiv
- 2611.04527
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.04527v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
