Video World Models · 2026-10-15
PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents
Siqi Yang, Yimu Wang, Ziqi Cai, Chengxuan Zhu, Shuchen Weng, Erwin Wu, Zhaofei Yu, Kaipeng Zhang, Boxin Shi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
PAWA is a dataset and framework for action-conditioned panoramic multiplayer world modeling in Minecraft. It pairs each player’s POV video with a world-stabilized equirectangular panorama and aligned actions, using a distributed CPU gameplay + GPU offline rendering pipeline that scales beyond 64 players. The dual-tower model jointly predicts POV and panorama with ray-aligned cross-attention and uses a quadratic-phase player encoding. The authors report PAWA improves POV prediction quality over prior multiplayer world models.
Why it matters
The authors target a missing supervision signal for multi-agent prediction: effects of other players outside a camera’s field of view. By providing synchronized POV+panorama with aligned actions, and by using ray-aligned cross-attention to align viewing directions across projections, PAWA aims to model shared-world dynamics and co-player interactions more faithfully than prior multiplayer models that only learn from perspective views.
Method
- PAWA dataset: for Minecraft multiplayer, each replay yields each player’s POV video, world-stabilized equirectangular panorama, and aligned actions; pipeline records on CPUs without graphics clients and renders offline on GPUs, scaling beyond 64 players.
- Dual-tower model: POV and panorama towers jointly predict both streams; they are coupled by bidirectional ray-aligned cross-attention that uses world-frame viewing rays to align tokens across projections.
- Quadratic-phase player encoding: replaces simplex-based rotary player codes with quadratic-phase codes, enlarging the identity pool from m+1 to m^2 at fixed embedding dimension.
Limitation
The paper text does not state any specific limitation of the method or dataset.
Abstract (from arXiv)
A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's current field of view. A point-of-view (POV) video captures such changes only after they enter its field of view, whereas panoramic video records the surroundings in every viewing direction and has served as an intermediate representation of the world state in recent world models. Learning this task therefore requires synchronized POV and panoramic videos of interacting players together with their actions, yet no existing dataset provides this combination in Minecraft, an established testbed in which players act independently in a shared, modifiable 3D world. We present PAWA, a dataset and framework for action-conditioned panoramic multiplayer world modeling. The dataset pairs each player's POV video with world-stabilized panoramic video and aligned actions. Its distributed pipeline decouples CPU gameplay capture from GPU rendering, scales beyond 64 simultaneous players, and re-renders each recorded session under new observation configurations without replaying the gameplay. On this dataset, we develop a dual-tower model that jointly predicts POV and panoramic video and couples the two streams through cross-attention that aligns tokens observing the same direction in a shared world frame. We further introduce a quadratic-phase player encoding that provides many more player codes than existing rotary codes at the same embedding dimension. Experiments show that PAWA outperforms state-of-the-art multiplayer world models in POV prediction quality while also producing high-fidelity panoramic predictions. The dataset and collection pipeline are open source to support further research on multiplayer world models.
Related papers
- WorldCast: Distributed Multiplayer World Models
- OverSeer: Multi-Agent World Models Beyond What Agents See
- ActionRoPE: action as coordinates for spatially consistent control in planar-camera game world models
- Multics: Generative Multi-Player World Model with Hierarchical Communication
- LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.20240
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's current field of view. A point-of-view (POV) video captures such changes only after they enter its field of view, whereas panoramic video records the surroundings in every viewing direction and has served as an intermediate representation of the world state in recent world models. Learning this task therefore requires synchronized POV and panoramic videos of interacting players together with their actions, yet no existing dataset provides this combination in Minecraft, an established testbed in which players act independently in a shared, modifiable 3D world. We present PAWA, a dataset and framework for action-conditioned panoramic multiplayer world modeling. The dataset pairs each player's POV video with world-stabilized panoramic video and aligned actions. Its distributed pipeline decouples CPU gameplay capture from GPU rendering, scales beyond 64 simultaneous players, and re-renders each recorded session under new observation configurations without replaying the gameplay. On this dataset, we develop a dual-tower model that jointly predicts POV and panoramic video and couples the two streams through cross-attention that aligns tokens observing the same direction in a shared world frame. We further introduce a quadratic-phase player encoding that provides many more player codes than existing rotary codes at the same embedding dimension. Experiments show that PAWA outperforms state-of-the-art multiplayer world models in POV prediction quality while also producing high-fidelity panoramic predictions. The dataset and collection pipeline are open source to support further research on multiplayer world models.