Driving · 2026-11-04

Future Representation Distillation for Imagination-free World Action Models

Jaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang

arXiv:2611.04527PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DreamFree is an imagination-free World Action Model that repurposes a pretrained video diffusion model as a predictive visual encoder via future representation distillation. A visual encoder predicts a frozen diffusion model’s representations from the current observation only, enabling action prediction without generating future videos. The authors report higher success rates than state-of-the-art VLA and WAM baselines and real-time closed-loop control at 11Hz with a 14B video backbone.

Why it matters

The work targets two deployment issues of video-generating WAMs: costly iterative video denoising and sensitivity of action prediction to inaccurate future visual forecasts. By distilling predictive knowledge into features computed from the current observation, DreamFree aims to deliver predictive priors for control without future video generation, while maintaining strong manipulation performance and real-time latency.

Method

  • Train a predictive visual encoder (initialized from a pretrained video diffusion backbone) to predict representations of a frozen video diffusion model using only the current observation and language instruction (future representation distillation).
  • Separate training into (1) predictive representation learning with distillation + native generative objective, and (2) action learning with an action module (readout + action decoder) trained from robot demonstrations.
  • During inference, run a single forward pass through the predictive visual encoder to extract action-relevant tokens; generate an action chunk without iterative video denoising (supports up to 11Hz for a 14B backbone).

Limitation

Scaling with embodied pretraining: the authors demonstrate a 14B video backbone trained on DROID without embodied pretraining can outperform existing VLA and WAM baselines, but RoboCasa results suggest additional video data could improve performance; they did not evaluate this extension on DROID due to computational constraints. They also note training complexity from separating backbone adaptation, predictive representation learning, and action learning, and propose joint training as future work.

Abstract (from arXiv)

World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC

preprint

Future Representation Distillation for Imagination-free World Action Models

Jaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang

First announced by arXiv: 2026-11-04 · Original record ↗

arxiv
2611.04527
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.04527v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.