Driving · 2026-11-06

PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State

Jihai Zhang, Yuhan Zhou, Junyou Li, Deheng Ye, Qi Dou, Yu Cheng

arXiv:2611.07867PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PlayGen2 is a real-time interactive world model that maintains explicit 3D state for persistent Doom gameplay. It separates learned 3D state evolution (a persistent volumetric occupancy grid plus identity-indexed entity states) from a neural renderer that synthesizes first-person RGB from projected depth/semantics and RGB history, without feeding generated images back into state prediction. The Doom prototype runs at ~160 ms backend latency and supports zero-shot unseen maps and composable multiplayer/interventions.

Why it matters

The authors aim to enable playable, controllable, persistent worlds by decoupling simulation state from neural rendering. Using generated frames alone can hinder long-term spatial/temporal consistency and make entity state hard to edit; PlayGen2 instead keeps an explicit 3D world memory that supports geometry-aware motion, player–NPC interactions, and extensible state-level composition and interventions.

Method

  • Maintain persistent world memory: static geometry as a volumetric occupancy grid plus identity-indexed 3D states for player and entities; update via a player-centered fixed-size 3D crop and write updates back beyond the local region.
  • Jointly learn action-conditioned entity dynamics, geometry completion, and entity discovery within the local region using separate geometry-generation (Discovery Transformer) and state-transition (Transition Transformer) branches.
  • Use a separately trained neural renderer that projects the predicted state into depth and semantic maps and synthesizes RGB from projected conditions plus RGB history; generated RGB never feeds back into world-state prediction.
Abstract (from arXiv)

Building a playable world requires more than generating plausible video: it demands responsive control, persistent spatial structure, and coherent interactions with dynamic entities. Using generated frames as the sole recurrent world representation ties these requirements to view-dependent appearance synthesis, making it difficult to maintain long-term spatial and temporal consistency in generation and interaction. We present PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering. PlayGen2 maintains a persistent world memory comprising static geometry represented by a volumetric occupancy grid and identity-indexed 3D states for the player and other entities. Within a fixed-size local region, its world-state model jointly learns action-conditioned entity dynamics, geometry completion, and entity discovery, committing updates to a persistent world memory. A separate neural renderer synthesizes first-person observations from depth and semantic projections of the world state, without feeding generated images back into state prediction. Using Doom as a testbed, we demonstrate that the model successfully learns world dynamics including geometry-aware player motion and player--NPC interactions. The framework is computationally efficient, enabling the complete simulation and rendering pipeline to achieve approximately 160ms backend latency on a MacBook Pro with an M4 Pro chip. Beyond single-player interaction, we demonstrate zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions. These capabilities demonstrate the transferability and extensibility of the learned game-world model. Code and pretrained model weights are publicly available.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC

preprint

PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State

Jihai Zhang, Yuhan Zhou, Junyou Li, Deheng Ye, Qi Dou, Yu Cheng

First announced by arXiv: 2026-11-06 · Original record ↗

arxiv
2611.07867
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07867v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Building a playable world requires more than generating plausible video: it demands responsive control, persistent spatial structure, and coherent interactions with dynamic entities. Using generated frames as the sole recurrent world representation ties these requirements to view-dependent appearance synthesis, making it difficult to maintain long-term spatial and temporal consistency in generation and interaction. We present PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering. PlayGen2 maintains a persistent world memory comprising static geometry represented by a volumetric occupancy grid and identity-indexed 3D states for the player and other entities. Within a fixed-size local region, its world-state model jointly learns action-conditioned entity dynamics, geometry completion, and entity discovery, committing updates to a persistent world memory. A separate neural renderer synthesizes first-person observations from depth and semantic projections of the world state, without feeding generated images back into state prediction. Using Doom as a testbed, we demonstrate that the model successfully learns world dynamics including geometry-aware player motion and player--NPC interactions. The framework is computationally efficient, enabling the complete simulation and rendering pipeline to achieve approximately 160ms backend latency on a MacBook Pro with an M4 Pro chip. Beyond single-player interaction, we demonstrate zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions. These capabilities demonstrate the transferability and extensibility of the learned game-world model. Code and pretrained model weights are publicly available.