Video World Models · 2026-10-15
StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys, Luc Van Gool
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
StateCraft decouples a persistent, updatable abstract state from the visual state in video world models. It pairs a language model that updates an explicit graph-based abstract state with a video diffusion model that generates the corresponding observations. Evaluated on generative software with a new benchmark and dataset pipeline, StateCraft improves visual memory metrics and supports state edits that guide generation beyond observed history, plus a proof-of-concept 3D case.
Why it matters
The authors address long-horizon consistency in video world models where actions can change out-of-view parts of the world. By maintaining an explicit abstract state (editable and updatable) rather than relying only on frame histories, they aim for consistent revisits, controllability under explicit state edits, and state-guided generation beyond what is visible in observations.
Method
- Factorizes latent state into a persistent abstract state (graph of entities/properties) and a visual state (current view), with actions updating the abstract state via sparse diffs.
- Couples a small pretrained VLM with low-rank finetuning to predict abstract-state transitions/diffs and a video diffusion model to generate next frames conditioned on the predicted abstract state and visual/action history.
- Develops an annotation-free, large-scale data collection pipeline for generative interfaces and a benchmark evaluating generation quality, memory (revisit consistency), and controllability under explicit abstract-state edits.
Limitation
In controllability tests, inserting new elements on empty background remains difficult (81% overall follow rate; insertion follow rate 0.48, with baseline follow 0.00).
Abstract (from arXiv)
Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.
Related papers
- Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
- WorldCast: Exploring Intelligent Visual Generation in World State Prediction
- Reading, Writing, Sharing, and Interacting with the World through Video
- State World Models: Execute the Rules, Generate the Pixels
- UniState: Unifying Memory and Intervention for Controllable World Simulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.19795
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.