Driving · 2026-11-06
PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State
Jihai Zhang, Yuhan Zhou, Junyou Li, Deheng Ye, Qi Dou, Yu Cheng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
PlayGen2 is a real-time interactive world model that maintains explicit 3D state for persistent Doom gameplay. It separates learned 3D state evolution (a persistent volumetric occupancy grid plus identity-indexed entity states) from a neural renderer that synthesizes first-person RGB from projected depth/semantics and RGB history, without feeding generated images back into state prediction. The Doom prototype runs at ~160 ms backend latency and supports zero-shot unseen maps and composable multiplayer/interventions.
Why it matters
The authors aim to enable playable, controllable, persistent worlds by decoupling simulation state from neural rendering. Using generated frames alone can hinder long-term spatial/temporal consistency and make entity state hard to edit; PlayGen2 instead keeps an explicit 3D world memory that supports geometry-aware motion, player–NPC interactions, and extensible state-level composition and interventions.
Method
- Maintain persistent world memory: static geometry as a volumetric occupancy grid plus identity-indexed 3D states for player and entities; update via a player-centered fixed-size 3D crop and write updates back beyond the local region.
- Jointly learn action-conditioned entity dynamics, geometry completion, and entity discovery within the local region using separate geometry-generation (Discovery Transformer) and state-transition (Transition Transformer) branches.
- Use a separately trained neural renderer that projects the predicted state into depth and semantic maps and synthesizes RGB from projected conditions plus RGB history; generated RGB never feeds back into world-state prediction.
Abstract (from arXiv)
Building a playable world requires more than generating plausible video: it demands responsive control, persistent spatial structure, and coherent interactions with dynamic entities. Using generated frames as the sole recurrent world representation ties these requirements to view-dependent appearance synthesis, making it difficult to maintain long-term spatial and temporal consistency in generation and interaction. We present PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering. PlayGen2 maintains a persistent world memory comprising static geometry represented by a volumetric occupancy grid and identity-indexed 3D states for the player and other entities. Within a fixed-size local region, its world-state model jointly learns action-conditioned entity dynamics, geometry completion, and entity discovery, committing updates to a persistent world memory. A separate neural renderer synthesizes first-person observations from depth and semantic projections of the world state, without feeding generated images back into state prediction. Using Doom as a testbed, we demonstrate that the model successfully learns world dynamics including geometry-aware player motion and player--NPC interactions. The framework is computationally efficient, enabling the complete simulation and rendering pipeline to achieve approximately 160ms backend latency on a MacBook Pro with an M4 Pro chip. Beyond single-player interaction, we demonstrate zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions. These capabilities demonstrate the transferability and extensibility of the learned game-world model. Code and pretrained model weights are publicly available.
Related papers
- BoxWorld: Let World Engines Control and Video Generators Render
- Wizard: Manipulate Any World in Real Time via Agentic Control
- AgentGarten: Code Worlds for Evolving Agents
- GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling
- CoCode-World: Shared 3D Proxy Representation for Multi-Player World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC
PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State
First announced by arXiv: 2026-11-06 · Original record ↗
- arxiv
- 2611.07867
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07867v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Building a playable world requires more than generating plausible video: it demands responsive control, persistent spatial structure, and coherent interactions with dynamic entities. Using generated frames as the sole recurrent world representation ties these requirements to view-dependent appearance synthesis, making it difficult to maintain long-term spatial and temporal consistency in generation and interaction. We present PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering. PlayGen2 maintains a persistent world memory comprising static geometry represented by a volumetric occupancy grid and identity-indexed 3D states for the player and other entities. Within a fixed-size local region, its world-state model jointly learns action-conditioned entity dynamics, geometry completion, and entity discovery, committing updates to a persistent world memory. A separate neural renderer synthesizes first-person observations from depth and semantic projections of the world state, without feeding generated images back into state prediction. Using Doom as a testbed, we demonstrate that the model successfully learns world dynamics including geometry-aware player motion and player--NPC interactions. The framework is computationally efficient, enabling the complete simulation and rendering pipeline to achieve approximately 160ms backend latency on a MacBook Pro with an M4 Pro chip. Beyond single-player interaction, we demonstrate zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions. These capabilities demonstrate the transferability and extensibility of the learned game-world model. Code and pretrained model weights are publicly available.
