Video World Models · 2026-10-20

State World Models: Execute the Rules, Generate the Pixels

Chan Yan Ki, Yanran Zhang, Wenzhao Zheng, Xiaofeng Wang, Zheng Zhu, Lei Chen, Jie Zhou, Jiwen Lu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Stateful Harness, which separates rule execution/state evolution from video rendering. It maintains evolving semantic/spatial state under fixed rules, renders each resolved event using a frozen video model, and uses the generated clip to update state for the next turn. They evaluate rule compliance with WorldState-Bench (50 counterfactual twin pairs with hidden state), where the full harness reaches 92% PairFollow.

Why it matters

The paper addresses a gap in action-conditioned video world models: navigation and visible responsiveness do not ensure that the rendered outcome follows persistent rules when outcomes depend on hidden state. By making rule execution authoritative before a frozen renderer, and by testing on counterfactual hidden-state pairs, the authors aim to measure whether state transitions are faithfully realized in pixels across turns.

Method

Stateful Harness keeps an evolving semantic ledger (resources, attributes, progress, terminal status) and an editable spatial scene, while rule programs define allowed action outcomes and required state changes.
At each turn, the runtime resolves the user-selected action against the current state/rules, compiles the settled event into a VisualScript instruction, renders with a frozen video model, and then uses VLM observations to update state via rule-constrained validation.
WorldState-Bench uses 50 counterfactual twin pairs (shared initial frame and action; different hidden values) and measures whether the rendered endings comply with the executable rule under a Change-to-Comply protocol.

Limitation

The box-based clay model provides a coarse representation of spatial layout; errors in instance recovery propagate to the preview, and the text instruction does not always prevent the renderer from reproducing the clay appearance. Articulated pose and fine appearance are determined by the video model, and the spatial pipeline operates offline.

Abstract (from arXiv)

Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3, specifying the required endings raises this rate from 32% to 64%, while the full harness reaches 92% with the same frozen model. These results support separating rule execution from video rendering for world interactions with hidden state.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC

preprint