Video World Models · 2026-10-09
What 30,000 Hours of Ego-centric Video Does Not Teach
Jiahua Dong, Anurag Bagchi, Yash Jangir, Muhammad Zubair Irshad, Sergey Zakharov, Martial Hebert, Homanga Bharadhwaj, Yu-Xiong Wang, Vitor Campagnolo Guizilini, Pavel Tokmakov
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors scale Cosmos 3 ego-centric video world models from 300 to 30,000 hours and directly measure interaction fidelity (agent hands/body and manipulated objects) on an out-of-distribution benchmark. Data scaling improves both but unevenly: agent fidelity saturates early and conditioning can reach its ceiling with far less data, while object fidelity remains far lower and improves slowly. They introduce object-centric adaptive noise scheduling to improve object dynamics, with a remaining gap.
Why it matters
The paper argues that downstream task metrics can hide where scaling helps or fails. By separating agent motion fidelity from action effects on objects using interaction-fidelity metrics, the authors find that scaling video data mainly benefits agent modeling, while faithful object interaction dynamics lag. This motivates training/conditioning changes (not just more data) and provides a way to study interaction learning limits.
Method
- Train Cosmos 3 video world models on nested ego-centric manipulation data budgets from 300 to 30,000 hours, with three model variants and with/without skeleton conditioning.
- Evaluate on an out-of-distribution benchmark from a separate recording campaign; measure SCS separately for agent (hands) and manipulated objects, using mask-based structural consistency and track-based fidelity.
- Introduce object-centric adaptive noise scheduling (dynamic-region supervision) that shifts supervision from overall scene appearance toward manipulated object dynamics; compare against other interaction-focused objectives reported as not improving at scale.
Limitation
The authors state that object fidelity remains far below agent fidelity and that closing the remaining gap will require more than data.
Abstract (from arXiv)
World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100x improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that shifts capacity from scene appearance toward object dynamics, improving object fidelity though a substantial gap remains. Finally, our conclusions transfer to downstream humanoid modeling. Overall, our results suggest that scaling ego-centric data brings agent modeling close to its limit while leaving its effects on the world far behind, and that closing this gap will depend on how models are trained, not only on how much data they see.
Related papers
- Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
- Ego4WAM: What Matters When Scaling Egocentric Human Data for Robot Learning?
- World Models' Last Exam in Physics
- Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC
What 30,000 Hours of Ego-centric Video Does Not Teach
Preprint posted: 2026-10-08 · Original record ↗
- arxiv
- 2610.12464
Explicit source versions (1)
- v1 · 2026-10-08
- Preprint available
Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100x improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that shifts capacity from scene appearance toward object dynamics, improving object fidelity though a substantial gap remains. Finally, our conclusions transfer to downstream humanoid modeling. Overall, our results suggest that scaling ego-centric data brings agent modeling close to its limit while leaving its effects on the world far behind, and that closing this gap will depend on how models are trained, not only on how much data they see.