Driving · 2026-11-03
W2A: Distilling World Priors into Action Models for Real-Time Robotic Control
Xiaopeng Zhang, Yongjin Mu, Yueyang Weng, Chengzhi Zhao, Yanjie Li
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
W2A is a two-stage distillation method that transfers world priors from a language-conditioned video diffusion transformer into an action model for real-time robotic control. During training, intermediate denoising video latents are compressed with PCA and distilled into an action DiT using a Q-Former. At test time, the video model is discarded, yielding video-free deployment with higher inference efficiency.
Why it matters
Most world action models require explicit future imagination at inference time, which adds overhead and latency that can prevent real-time robotic control. The authors’ approach removes the video model from the deployment graph while retaining performance, aiming to make video-based world priors usable for real-time manipulation.
Method
- Stage I: adapt a pretrained language-conditioned video DiT (Cosmos-Predict2) to robotic videos with LoRA, extract intermediate denoising latents, and compress them by fitting PCA and projecting latents into a lower-dimensional subspace.
- Stage II: train an action DiT from scratch with a flow matching objective plus a distillation loss that aligns Q-Former queried action latents to the compressed PCA-projected video latents.
- Inference: discard the video DiT entirely; predict action chunks with the action DiT, execute only the first action subset, and warmstart replanning by reusing the unexecuted suffix perturbed to an intermediate flow time.
Limitation
The authors report that compositional generalization is bounded by the support of the training distribution: W2A may fail on entirely unseen tasks/objects and when the initial scene configuration deviates substantially from training, and it does not retain explicit online future imagination or strong recovery mechanisms for out-of-distribution situations.
Abstract (from arXiv)
World Action Models (WAMs) have recently shown great promise in robotic control, benefiting from the future imagination capability of large-scale pretrained video models. However, most existing WAMs require explicitly imagining future states at test time, incurring substantial computational overhead and latency that make them unsuitable for real-time robotic control. In this paper, we propose W2A, a two-stage distillation framework for real-time robotic manipulation that distills intermediate denoising representations from a video diffusion model into an action model during training, while entirely discarding the video model at test time. We hypothesize that although video latent representations are high-dimensional, a compressed low-dimensional subspace is sufficient to provide effective supervision for action generation. This makes it possible to transfer such information effectively into an action model. Extensive experiments in both simulation and real-world settings demonstrate competitive success rates while significantly improving inference efficiency. On LIBERO, W2A achieves a 95.9% average success rate. More importantly, W2A achieves about 21x faster inference than the representative WAM baseline.
Related papers
- Self-Distilling Future Video Representations for World Action Models
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
- Future Representation Distillation for Imagination-free World Action Models
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
- Box-WAM: Box Trajectories as Compact State Representations for World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-02 · Indexed by Pub-AI: 2026-11-03 UTC
W2A: Distilling World Priors into Action Models for Real-Time Robotic Control
Preprint posted: 2026-11-02 · Original record ↗
- arxiv
- 2611.02600
Explicit source versions (1)
- v1 · 2026-11-02
- Preprint available
Source date: 2026-11-02 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.02600v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World Action Models (WAMs) have recently shown great promise in robotic control, benefiting from the future imagination capability of large-scale pretrained video models. However, most existing WAMs require explicitly imagining future states at test time, incurring substantial computational overhead and latency that make them unsuitable for real-time robotic control. In this paper, we propose W2A, a two-stage distillation framework for real-time robotic manipulation that distills intermediate denoising representations from a video diffusion model into an action model during training, while entirely discarding the video model at test time. We hypothesize that although video latent representations are high-dimensional, a compressed low-dimensional subspace is sufficient to provide effective supervision for action generation. This makes it possible to transfer such information effectively into an action model. Extensive experiments in both simulation and real-world settings demonstrate competitive success rates while significantly improving inference efficiency. On LIBERO, W2A achieves a 95.9% average success rate. More importantly, W2A achieves about 21x faster inference than the representative WAM baseline.
