Video World Models · 2026-10-02
Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
Xindi Yang, Baolu Li, Liam Lee, Zhenfei Yin, Songxin Zhang, Zhuoyang Song, Xu Jia, Jianfei Cai, Tien-Tsin Wong, Bingyi Jing, Mengyue Yang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Oneira is an interactive video world model that maintains an explicit, extensible world state table updated by a coding agent. The agent grounds actions to object instances, plans interactions, updates object presence and interaction state (e.g., removal/recoloring), and extends the table with newly generated objects during exploration. The updated table is rendered into a coarse conditioning video for a pretrained video generator, enabling open-world interactivity and persistent state over long horizons.
Why it matters
The authors address a gap in generative video world models: open-ended generation may expand what the agent sees without expanding what it can act on, and action effects may not persist beyond transient pixels/latent context. Oneira separates explicit world evolution (a state table) from visual realization (video generation), aiming to make newly created entities actionable and interaction outcomes durable across segments.
Method
- A coding agent continuously manages an explicit, extensible world state table (objects’ 3D boxes/pose, presence, and interaction state) to support Open-World Interactivity and Persistent State.
- For each segment, the agent reads the current state and goal/action, grounds intent to a target instance and execution time, plans a sequence, and writes deterministic local state transitions to the table.
- The updated table is rendered along the camera trajectory into a coarse conditioning video (with colored boxes encoding object states); a pretrained video generator fills appearance/motion details under the learned state-aligned conditions.
Abstract (from arXiv)
Generative video world models can now synthesize open-ended environments that agents can navigate and interact with in simple ways. Yet open-ended generation does not imply full interaction: as a generated world expands, newly created content through navigation should expand what the agent can act upon, and as the agent changes the world, those changes should become persistent parts of the environment rather than transient visual effects. We characterize these two requirements as Open-World Interactivity, where newly generated or encountered entities are incorporated into the actionable world, and Persistent State, where interaction outcomes are committed to the world state and continue to influence subsequent observations and interactions. We present Oneira, an interactive video world model that closes the loop between generation and interaction through an explicit, extensible world state managed by a coding agent. Given the current observation and an action or high-level goal, the agent reads the world state, grounds the relevant entities, plans the interaction, and writes its outcome back into a world state table. When exploration reveals new objects, the agent incorporates them from generated observations, allowing the interaction space to expand with the generated world. Meanwhile, previously induced state changes are carried across video segments, making the consequences of interaction persistent parts of subsequent world evolution. The updated world state is rendered along the camera action trajectory into a coarse conditioning video, from which a video generator fills in the appearance, motion, and interaction details not represented in the state. Experiments show that Oneira enables direct and consistent interaction with newly generated objects, while preserving the effects of prior interactions over long horizons. Project page: https://madaoer.github.io/projects/oneira
Related papers
- StateCraft: Augmenting Video World Models with Abstract-State Reasoning
- Reading, Writing, Sharing, and Interacting with the World through Video
- Generative World Simulation with DAO and JING
- UniState: Unifying Memory and Intervention for Controllable World Simulation
- World Observer: Joint Actor-Observer Generation for Persistent World Modeling
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC
Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.01614
Explicit source versions (1)
- v1 · 2026-10-01
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Generative video world models can now synthesize open-ended environments that agents can navigate and interact with in simple ways. Yet open-ended generation does not imply full interaction: as a generated world expands, newly created content through navigation should expand what the agent can act upon, and as the agent changes the world, those changes should become persistent parts of the environment rather than transient visual effects. We characterize these two requirements as Open-World Interactivity, where newly generated or encountered entities are incorporated into the actionable world, and Persistent State, where interaction outcomes are committed to the world state and continue to influence subsequent observations and interactions. We present Oneira, an interactive video world model that closes the loop between generation and interaction through an explicit, extensible world state managed by a coding agent. Given the current observation and an action or high-level goal, the agent reads the world state, grounds the relevant entities, plans the interaction, and writes its outcome back into a world state table. When exploration reveals new objects, the agent incorporates them from generated observations, allowing the interaction space to expand with the generated world. Meanwhile, previously induced state changes are carried across video segments, making the consequences of interaction persistent parts of subsequent world evolution. The updated world state is rendered along the camera action trajectory into a coarse conditioning video, from which a video generator fills in the appearance, motion, and interaction details not represented in the state. Experiments show that Oneira enables direct and consistent interaction with newly generated objects, while preserving the effects of prior interactions over long horizons. Project page: https://madaoer.github.io/projects/oneira