Video World Models · 2026-10-06

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou, Lingjun Mao, Ruobing Han, Jinglin Cao, Wenpeng Xu, Lukun He, Minghao Fu, Fan Feng, Biwei Huang

arXiv:2610.04432 PDF Code

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Video2World studies autonomous video-to-simulation as an end-to-end software engineering task: a coding agent observes an embodied RGB video, constructs a simulated environment and robot behavior, then iteratively refines using execution feedback. The benchmark contains 222 reconstruction instances from 189 robot/human demonstration videos, evaluated with geometric fidelity, dynamic fidelity, and functional correctness.

Why it matters

The authors benchmark whether frontier coding agents can integrate spatial perception, physical reasoning, and executable coding to produce interactive simulator worlds from embodied video. They report strong variation across metrics and agents, and show that higher visual/geometric fidelity does not reliably imply higher task success.

Method

Formulate autonomous video-to-simulation: an agent outputs (η,π) to define reconstructed environment and robot behavior from an RGB embodied video, then revises via execution feedback within a fixed budget.
Evaluate reconstructions with executability/build rate, geometric fidelity (scene/object surface/shape/pose), dynamic fidelity (object motion discrepancy), and functional correctness (task success and graded progress).
Run 9 frontier coding-agent systems on 222 instances under the same benchmark interface and assigned target simulator/robot/control configuration. Retain submissions and execution traces for reproducibility.

Limitation

Functional correctness is measured from joint execution of the reconstructed environment and the agent-generated robot behavior; a failed rollout may originate from errors in the reconstructed world, the behavior policy, or their interaction.

Abstract (from arXiv)

Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate \emph{autonomous video-to-simulation} as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce \textbf{Video2World}, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5\% to over 15\%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work worse: better visual fidelity does not always lead to higher task success. This echoes the broader gap between perceptual realism and factual correctness observed in generative models.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC

preprint