Latent Dynamics · 2026-10-09

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

Zheyu Fan, Yue Zhang, Mingkai Deng, Kangrui Wang, Qineng Wang, Canyu Chen, Jie Hao, Xing Fan, Chenlei Guo, Eric P. Xing, Mohit Bansal, Manling Li

arXiv:2610.12417 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

WOVEN introduces a training source and benchmark for visual transition reasoning in multimodal LLMs. It generates (s,a,s′) transitions using a video-pretrained generative model, labeling items by 20 scene types, 5 action types, and 8 reasoning types (36,076 items total). Evaluating 38 frontier MLLMs, the authors report a systematic deficit versus humans. Post-training MLLMs on WOVEN learn a transfer-capable shared capability and obtain gains on multiple external transition benchmarks.

Why it matters

The paper targets failures in spatial/embodied/physical/temporal reasoning by hypothesizing a shared deficit in action-conditioned visual transition reasoning. WOVEN provides controlled supervision organized by scene, action, and reasoning operation, enabling comparisons of which supervision transfers and a training recipe for visual world modeling that is validated prospectively on held-out benchmarks.

Method

Define visual transition reasoning over (s,a,s′) and formalize eight reasoning operations in four families (causal, counterfactual, physical, temporal). Build a taxonomy that varies supervision by scene, action, and reasoning type.
Generate controlled rollouts with a video generation model from specified initial frames and action descriptions, including counterfactual rollouts from the same initial state under different actions; convert rollouts into multiple-choice items with error-typed distractors.
Train MLLMs with SFT and GRPO on the full WOVEN set or controlled subsets (~2,000 items each), then evaluate on WOVEN test splits and zero-shot on 26 external benchmarks; use prospective tests to validate a training recipe.

Limitation

WOVEN scopes to world modeling, single-step, vision-grounded, action-conditioned reasoning over (s_t, a, s_{t+1}) triples, and therefore does not directly evaluate longer-horizon procedural planning, symbolic-environment dynamics inference, or purely latent-space predictive representations evaluated by downstream control.

Abstract (from arXiv)

Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect a shared deficit in visual transition reasoning, and test whether this capability can serve as a shared training primitive, one that different models can learn from different supervision sources and reuse across different tasks, with a systematic training recipe. Existing benchmarks document these deficits separately but do not support controlled comparisons across scenes, actions, and reasoning operations. We therefore introduce WOVEN, a training source and benchmark for visual transition reasoning that organizes transition supervision by scene, action, and reasoning type, using diverse, realistic rollouts from video-pretrained generative models: 36,076 examples across 20 scene types, 5 action types, and 8 reasoning types. We first evaluate 38 frontier MLLMs (e.g., GPT-5.4 and Qwen3-VL-235B-A22B) and find a substantial and systematic deficit: even the strongest models fall far below humans, and the failures recur across model families and persist with scale. We then train MLLMs at multiple scales on WOVEN and find that they learn a shared capability that transfers broadly: training subsets of only about 2,000 items each collectively improve 22 of 26 external benchmarks by up to 27.3 percentage points, and WOVEN data can replace 30-50% of a task's own training data with comparable accuracy. Controlled comparisons further yield a training recipe for visual world modeling, validated prospectively on held-out benchmarks: select supervision by the reasoning operation it teaches rather than by the actions, scenes, or domains it shows, and prefer larger changes to the visual state for robustness. Our work establishes visual transition reasoning as a reusable foundation for systematic visual world-model training in MLLMs.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC

preprint