Video World Models · 2026-10-21

Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directions

Zhiting Mei, Tenny Yin, Ola Shorinwa, Apurva Badithela, Zhonghe Zheng, Joseph Bruno, Madison Bland, Lihan Zha, Asher Hancock, Jaime Fern\'andez Fisac, Philip Dames, Anirudha Majumdar

arXiv:2601.07823 PDF Code

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

This survey reviews video world models for robotics, focusing on applications (robot data generation and imitation learning, dynamics/rewards modeling in RL, policy evaluation, and visual planning), key video-model architectures, and trustworthiness challenges. The authors highlight that video models can improve photorealistic, physics-consistent prediction, but still hallucinate (e.g., physics violations) and can struggle with instruction following, while facing data and compute overhead.

Why it matters

Robotic systems need world models that support policy learning, evaluation, and planning without prohibitive real-world costs. The paper argues video models can provide fine-grained embodied prediction and scalable data generation, but also emphasizes reliability and safety barriers—especially hallucinations, instruction-following failures, and content/interaction safety—needed for safety-critical robotics.

Method

The authors survey literature on video generation with emphasis on robotics applications, checking online sources, paper references, and individual suggestions, and aggregating publicly accessible papers up to December 2025.
They categorize papers into foundational methods (diffusion/flow-matching), application-centric solutions (robot policy learning, planning, and policy evaluation), and benchmarks/datasets (metrics and evaluation).
They include all representative robotics application papers and benchmark/dataset papers, and provide a curated bibliography at https://github.com/irom-princeton/awesome-robotics-video-world-model-papers .

Limitation

The paper is a survey and does not present new benchmark results; it also notes “significant data curation, training, and inference costs and the lack of adequate content safeguards in video generation remain critical challenges.”

Abstract (from arXiv)

Video world models have emerged as promising candidates for high-fidelity world models, offering the potential to synthesize high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models can generate photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. However, despite their potential, video models still generate physics-violating future predictions, often manifesting as hallucinations. In this survey, we provide a review of video models and their applications as embodied world models in robotics, including efficient data generation and policy learning, dynamics and rewards modeling in reinforcement learning, policy evaluation, and visual planning. Further, we highlight important challenges hindering the trustworthy integration of video models, such as poor instruction following, hallucinations like violations of physics, unsafe content generation, in addition to significant data and compute overhead. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings. We provide a curated bibliography at https://github.com/irom-princeton/awesome-robotics-video-world-model-papers .

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-01-12 · Indexed by Pub-AI: 2026-10-21 UTC

preprint