Video World Models · 2026-10-28
BoxWorld: Let World Engines Control and Video Generators Render
Zike Wu, Ahmed Hatem, Chintan Trivedi, Wallace Michel Pinto Lira, Xuebin Qin, Lele Wang, Renjie Liao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
BoxWorld separates interaction reasoning from generative rendering for real-time interactive video worlds with user-defined behaviors. A coding agent builds a Box World Engine that updates object states via a minimal Object-Centric State Interface (layout boxes, persistent attributes, dynamic status). A Box-State Generative Renderer then autoregressively renders the evolving state stream, trained with a three-stage strategy and evaluated with PairedStateBench plus video-quality benchmarks.
Why it matters
The authors target controllable interactive video worlds where object-level rules can be specified and their consequences rendered online. By using explicit object-centric state updates outside the generator, BoxWorld aims to make rule-based outcomes easier to specify/control while keeping real-time rendering and coherent video generation.
Method
- Box World Engine: maintains internal scene variables and applies user-specified rules to update object layout boxes, persistent attributes, and dynamic status variables published through the Object-Centric State Interface.
- Box-State Generative Renderer: autoregressively generates latent video while injecting states frame-by-frame using a State Grounding Module (masked cross-attention using object boxes) and async streaming between engine and renderer.
- Three-stage training: (1) Masked Layout Grounding on open-domain videos with Diffusion Forcing; (2) State Dynamics Adaptation on gameplay videos with teacher forcing and LoRA adapters; (3) Rollout Distillation using Self-Forcing-style distribution matching (DMD) on self-generated rollouts.
Limitation
Current limitations mainly arise from the interface, the engine, and the training annotations. The Object-Centric State Interface may be insufficient for highly complex cases involving fine-grained geometry, articulated motion, or contact-rich interactions; rendered consequences depend on the engine; and the renderer is trained on automatically annotated videos where occasional annotation errors (e.g., missed tracking) may lead to missing consequences or identity swaps.
Abstract (from arXiv)
We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.
Related papers
- PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State
- StateCraft: Augmenting Video World Models with Abstract-State Reasoning
- Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
- ReactWorld: Structuring Interaction Responses for Embodied Video Generation
- WorldDraft: If Code Builds the World, Can Video Models Just Render It?
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-26 · Indexed by Pub-AI: 2026-10-28 UTC
BoxWorld: Let World Engines Control and Video Generators Render
Preprint posted: 2026-10-26 · Original record ↗
- arxiv
- 2610.36371
Explicit source versions (1)
- v1 · 2026-10-26
- Preprint available
Source date: 2026-10-26 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.36371v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.
