Driving · 2026-10-14
What Limits Cross-Task Generalization in World Action Models?
Chen Zhao, Zipeng Ye, Yihan Lin, Yuchen Xie, Jing Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors show that cross-task generalization fails for world action models (WAMs) because decoded actions from world-model-predicted futures collapse, unlike actions recovered from recorded futures. They introduce AoE (a frozen, camera-space inverse-dynamics model) to evaluate and supervise. Using VITA loss (AoE-guided action discrepancy), execution success of generated futures increases from near zero to 47.0% across two world models and three manipulation benchmarks.
Why it matters
World-model evaluations in video space may miss whether predicted futures contain action-relevant information that transfers across tasks/environments. By decoupling world modeling from action recovery with AoE and introducing VITA loss, the work links prediction quality to downstream executability and provides a direct training signal to target action-critical regions (e.g., gripper geometry).
Method
- Decouple a WAM into a world model and a pretrained inverse-dynamics model (AoE), then keep the inverse-dynamics model frozen while swapping recorded future images with world-model predictions to evaluate executability of generated futures.
- Introduce AoE, a camera-space inverse-dynamics model pretrained on 9,637 hours of robot image–action pairs; its camera-space action representation supports zero-shot action recovery across environments by handling coordinate conversion at execution.
- Use VITA loss: pass predicted futures through frozen AoE and penalize discrepancies in recovered action-space (action-flow representation), redirecting future generation toward action-relevant visual details.
Abstract (from arXiv)
Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce Align Once, Evaluate Everywhere (AoE), a camera-space inverse-dynamics model pretrained on large-scale robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the vision-to-action (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.
Related papers
- OpenWAM: An Open Framework for Composable World-Action Models
- The Planning Limits of Latent World Models
- Plausible Future, Wrong Choice: Steering World-Action Models toward Intended Future
- Achieve What You Imagined: Learning to Align Actions with Visual Plans
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-14 UTC
What Limits Cross-Task Generalization in World Action Models?
Preprint posted: 2026-10-12 · Original record ↗
- arxiv
- 2610.16641
Explicit source versions (1)
- v1 · 2026-10-12
- Preprint available
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce Align Once, Evaluate Everywhere (AoE), a camera-space inverse-dynamics model pretrained on large-scale robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the vision-to-action (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.