Robotics · 2026-10-16
Can World and Action Modeling Benefit Each Other under Unified Causal Training?
Yue Su, Bokai Lin, Xinkai Wang, Chubin Zhang, Xiangnan Wu, Kuan Zhang, Weiyang Jin, Jiwen Yu, Yiming Li, Cewu Lu, Lixin Yang, Xihui Liu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce OneWM, a unified causal training framework that uses a shared video–action architecture where actions are both conditions for forward dynamics prediction and targets for action generation. On LIBERO, LIBERO-Plus, and RoboCasa365 (plus real-world tasks), they report that learning action-conditioned forward dynamics improves policy performance and generalization, and that learning to generate actions improves action-conditioned multi-view world prediction.
Why it matters
Unified causal training that couples action-conditioned world modeling and action generation could enable reciprocal improvements between learning dynamics and learning control. The paper tests this within a shared causal mixture-of-transformers design and reports improvements in both policy learning and action-conditioned video prediction.
Method
- OneWM trains a shared causal MoT backbone with separate video and action experts connected by temporally causal joint attention, conditioning on language and using a video VAE.
- Actions are unified via a clean-prefix length r: all-clean (actions condition future video), partial-clean (clean prefix conditions video and action denoising on noisy suffix), and all-noisy (actions are denoising targets).
- Training uses flow matching with a joint video and action objective; inference uses all-clean sampling for world prediction and all-noisy sampling for action generation (closed-loop replanning).
Limitation
The paper notes that LIBERO, LIBERO-Plus, and RoboCasa365 are designed for policy evaluation and do not provide dedicated validation sets for action-conditioned video prediction; the authors therefore construct evaluation video sets from expert rollouts.
Abstract (from arXiv)
World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.
Related papers
- OpenWAM: An Open Framework for Composable World-Action Models
- Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation
- PrismWAM: Decoupling State Representations in World Action Models
- OneLA: Learning Many Worlds from Latent Actions
- Rethinking Representations for World-Action Modeling
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC
Can World and Action Modeling Benefit Each Other under Unified Causal Training?
Preprint posted: 2026-10-15 · Original record ↗
- arxiv
- 2610.21095
Explicit source versions (1)
- v1 · 2026-10-15
- Preprint available
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.