Driving · 2026-10-08
Video Prediction Policy 2: Predict Better, Act Better
Yanjiang Guo, Haodong Yan, Zhide Zhong, Zhongru Zhang, Qingyuan Yang, Qingzhou Lu, Xiaoyu Chen, Yen-Jen Wang, Shuying Deng, Chenghan Yang, Puzhen Yuan, Chenxin Liu, Tun Ban, Xiang Zhu, Yichen Liu, Kun Feng, Haoang Li, Jianyu Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Video Prediction Policy 2 (VPP2) is a world action model that targets open-ended manipulation failures of prior WAMs caused by (1) video models not being optimized for manipulation and (2) naively adding action components degrading generalization. VPP2 continues pretraining on manipulation videos with detailed captions and event-level video training, post-trains/distills to a single-step visual planner, and trains an MoT action expert for inverse dynamics.
Why it matters
The authors report that VPP2 improves zero-shot and post-trained manipulation generalization by strengthening instruction-following video prediction and pairing it with action learning. They present quantitative gains on video instruction-following and multiple robot benchmarks (real-world ALOHA, LIBERO-Pro/OOD, and RoboDojo).
Method
- Continues pretraining a video foundation model on curated manipulation videos using detailed captions and event-level video prediction training to encourage consistent semantic-to-trajectory mappings.
- Post-trains and distills the video model to predict fixed-horizon future chunks (2s human hand, 8s robot manipulation) and generate single-step predictions for a real-time visual planner.
- Trains an action expert (0.9B DiT with MoT) to generate 2-second action chunks conditioned on one-step video latents, with LoRA-based adaptation and a diffusion transformer-based inverse dynamics learning approach.
Abstract (from arXiv)
World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limitation to two factors: (1) base video models are not optimized for manipulation, and (2) naively incorporating action components into video models can substantially degrade their generalization capabilities. We introduce Video Prediction Policy 2 (VPP2), a WAM that enables strong zero-shot generalization in both video prediction and action generation. First, we curate a large-scale, diverse dataset of manipulation videos to continue pretraining the base video foundation model. We annotate video clips with detailed captions and perform \textit{event-level} video pretraining to promote generalization across open-ended manipulation tasks. Second, we post-train and distill the video model into a single-step visual planner with fixed prediction horizon. Finally, we introduce action module via a mixture-of-transformers (MoT) architecture to learn implicit inverse dynamics model. Experiments demonstrate three key results: (1) VPP2-14B outperforms Cosmos3-64B by 11.0\% points in video prediction instruction-following success rate on open-ended tasks; (2) VPP2 surpasses the strongest baseline by 18.5\% points in success rate on real-world zero-shot ALOHA manipulation tasks; and (3) following benchmark-specific post-training, VPP2 achieves the highest success rates among evaluated methods on the challenging LIBERO-Pro, LIBERO-OOD, and RoboDojo benchmarks.
Related papers
- WorldBridge: Bridging Video and Action for Task Generalization in World Action Models
- Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction
- OpenWAM: An Open Framework for Composable World-Action Models
- Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
- Self-Distilling Future Video Representations for World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC
Video Prediction Policy 2: Predict Better, Act Better
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.10270
Explicit source versions (1)
- v3 · 2026-10-13
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v3
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limitation to two factors: (1) base video models are not optimized for manipulation, and (2) naively incorporating action components into video models can substantially degrade their generalization capabilities. We introduce Video Prediction Policy 2 (VPP2), a WAM that enables strong zero-shot generalization in both video prediction and action generation. First, we curate a large-scale, diverse dataset of manipulation videos to continue pretraining the base video foundation model. We annotate video clips with detailed captions and perform \textit{event-level} video pretraining to promote generalization across open-ended manipulation tasks. Second, we post-train and distill the video model into a single-step visual planner with fixed prediction horizon. Finally, we introduce action module via a mixture-of-transformers (MoT) architecture to learn implicit inverse dynamics model. Experiments demonstrate three key results: (1) VPP2-14B outperforms Cosmos3-64B by 11.0\% points in video prediction instruction-following success rate on open-ended tasks; (2) VPP2 surpasses the strongest baseline by 18.5\% points in success rate on real-world zero-shot ALOHA manipulation tasks; and (3) following benchmark-specific post-training, VPP2 achieves the highest success rates among evaluated methods on the challenging LIBERO-Pro, LIBERO-OOD, and RoboDojo benchmarks.