Latent Dynamics · 2026-10-12
What Visual Representations Do World Action Models Need?
Kai Liu, Jiangran Lyu, Tingyu Deng, Yixin Zheng, Jingtian Wan, Wenbo Cui, Mi Yan, Yizhou Zhou, Haoran Li, Dongbin Zhao, Zhizheng Zhang, He Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
World Action Models (WAMs) use VAE representations learned for visual reconstruction rather than manipulation. The authors run controlled comparisons of visual representations, five feature-injection mechanisms, and feature combinations, and instantiate the best “Representation Recipe” as R2-WAM. They report robustness gains from general-purpose representations, highest success with channel-wise joint prediction, and limited additional gains from composing more sources beyond the first improvement.
Why it matters
The paper isolates how much of WAM performance comes from (i) which visual representation is used, (ii) how external features are integrated, and (iii) whether adding more sources helps. The authors provide empirical guidance for selecting and integrating pretrained visual encoders into WAMs, and demonstrate a concrete recipe (R2-WAM) validated on multiple formulations and real-robot tasks.
Method
- Use a common WAM backbone (pretrained video expert + separate action expert) with frozen external visual encoders; compare visual representations, five integration mechanisms, and feature compositions under shared training protocol.
- Evaluate representations in LIBERO-Plus with channel-wise concatenation over seven perturbation categories, using VAE-only WAM as baseline.
- Compare integration mechanisms (including joint prediction variants) and study progressive feature composition starting from DINOv3, then instantiate the resulting recipe as R2-WAM and validate on RoboCasa GR1 and RoboTwin 2.0.
Limitation
The authors state that “performance is near saturation” for LIBERO evaluation, and full results are provided in the Appendix rather than in the main text.
Abstract (from arXiv)
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R$^2$-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.
Related papers
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- PICA-WAM: Visual Perturbation Invariance via Cross-Appearance Self-Alignment in World Action Models
- Self-Distilling Future Video Representations for World Action Models
- World Action Modeling with Progressive Visual Planning
- How Representational Priors Shape Generalization in World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC
What Visual Representations Do World Action Models Need?
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.13024
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R$^2$-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.