Video World Models · 2026-10-23

Reading, Writing, Sharing, and Interacting with the World through Video

Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

This survey frames video foundation models and video world models using a shared world-modeling lens: an observed video is a partial, lossy, temporally ordered projection of a latent temporal world. It organizes the literature into four operations—reading, writing, sharing, and interacting—and distinguishes temporal coherence of traces from consistency of the implied world state, with emphasis shifting toward persistent state, controllable evolution, and interaction-based evaluation.

Why it matters

The authors propose a unifying framework for comparing diverse video model goals—understanding, generation, unified systems, and world prediction/control. By separating temporal coherence from world consistency and by organizing work into four operations, the survey clarifies how models represent state, how control and actions are incorporated, and what evaluation should test for persistent, physically/causally valid world dynamics rather than only frame-level plausibility.

Method

View an observed video as a partial, lossy, temporally ordered trace of a latent temporal world; define represented state and two forms of consistency.
Organize video world/model literature into four operations: reading (infer world properties), writing (synthesize video), sharing (integrate both), interacting (update/predict world states under actions and evolving observations).
Use an evidence ladder and a distinction between temporal coherence and world consistency to motivate interaction- and decision-grounded evaluation.

Limitation

Not stated in the provided text.

Abstract (from arXiv)

Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC

preprint