Driving · 2026-10-22
Agentic World Action Models
Zhuoyang Liu, Dantong Niu, Botong Zhang, Boning Shao, Jing Wang, Ruijie Zheng, Yuvan Sharma, Runyu Lu, Jiaming Liu, Renrui Zhang, Roei Herzig, Yutong Bai, Ken Goldberg, Danfei Xu, Yuke Zhu, Guanzhi Wang, Linxi "Jim" Fan, Trevor Darrell
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AW-0 is an agentic world action model that integrates code-based planning with joint video and action prediction. The authors jointly generate code (skill programs/logic) and future robot behavior as three coupled streams in a mixture-of-transformers architecture. An automated curation pipeline labels demonstrations with aligned programs and consolidates routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual tasks show higher task performance than VLA and WAM baselines.
Why it matters
The paper targets a gap in world action models: predictions of future video/actions do not specify how to decompose long-horizon tasks into reusable, compositional skills. By adding a grounded code expert and training it jointly with future visual dynamics and continuous control, AW-0 aims to connect planning, prediction, and execution, including hybrid execution that can improve robustness under distribution shift.
Method
- AW-0 formulates unified conditional generative modeling over discrete skill code, continuous future video latents, and continuous action chunks, conditioned on current multi-view observations, language instruction, robot state, and program memory.
- It uses a mixture-of-transformers with three experts (code, video, action) that jointly denoise code via discrete masked diffusion and video/action via continuous flow matching, with cross-modal conditioning via masked joint attention.
- It introduces an automatic annotation pipeline that samples episodes, uses Gemini 3.1 Pro and Claude Code/Opus to generate and distill subtask programs into a frozen skill library, then re-annotates remaining data by cached reuse or generating new labels.
Limitation
The authors state that generated programs can select appropriate skills yet contain incorrect arguments; syntactic validity does not ensure correct execution. They also state that planning improves WAM performance but does not eliminate sensitivity to visual distribution shifts, and hybrid execution trades clean performance for robustness under distribution shift. Real-world hybrid evaluation is limited to the YAM platform due to lack of code-as-policy systems for dexterous hand and the system does not learn online from its own interactions.
Abstract (from arXiv)
Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.
Related papers
- LAW: Unifying Language, Action, and World Modeling for Agentic Physical Intelligence
- WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
- Box-WAM: Box Trajectories as Compact State Representations for World Action Models
- Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation
- Magic-W0: A Structured World-Action Foundation Model for Physical Intelligence
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC
Agentic World Action Models
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.30136
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.