Driving · 2026-10-14
Learning Transferable World-Action Models from Task-Paired Human Videos
Jeongeun Park, Taekyung Kim, Dongyoon Han, Sangdoo Yun
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
WATCH is a cross-video world-action pre-training from task-paired human videos. Conditioned on one demonstration’s video and actions, the model predicts another demonstration’s actions and future frames from the query’s current observation. The authors report improved motion alignment across embodiments and better action/video prediction on unseen robots, and higher MimicDroid success after robot post-training.
Why it matters
Standard within-video next-frame objectives can entangle predictions with demonstration-specific appearance. By predicting across demonstrations with different scenes and embodiments, WATCH aims to learn task-relevant interaction representations that transfer to new robots, without requiring matched human-robot demonstrations for pre-training.
Method
- Cross-video pre-training: condition on a reference human demonstration (video + actions) and predict a query demonstration’s actions and future frames from the query’s current frame and state.
- Human-human pairing across sources/scenes: build query–reference pairs from different EgoVerse episodes with a task taxonomy and dynamic-time-warping-based selection.
- Robot post-training for control: after WATCH, post-train the same model for in-context (reference provided) and context-free (reference removed) robot action/video prediction.
Limitation
The authors note a caveat for their held-out-embodiment regression: WATCH receives the reference’s future actions as input, which bears directly on a future-motion probe.
Abstract (from arXiv)
World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.
Related papers
- Native Action-Prior Learning from Videos for World Action Models
- Being-M0.7: A Latent World-Action Model for Humanoid Robots
- Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
- DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
- OpenWAM: An Open Framework for Composable World-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-14 UTC
Learning Transferable World-Action Models from Task-Paired Human Videos
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.14351
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.