Video World Models · 2026-10-30
Wizard: Manipulate Any World in Real Time via Agentic Control
Yuzhe Wu, Zijun Lin, Yeying Jin
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Wizard is an agentic game world model for real-time, precise character motion control with zero-shot generalization across novel scenes, characters, and actions. It uses structured geometric proxies (scene geometry/depth-height) and articulated skeletons, plus an adaptive masking mechanism that gates reference attention to corresponding character regions. An agent maintains explicit game states and updates them for mechanics-consistent combat during closed-loop play at 24 FPS.
Why it matters
The authors target interactive controllability in game-like settings: fine-grained motion control, persistent state/game rules outside the video generator, and zero-shot adaptation to new scenes and characters. Wizard’s structured proxy conditioning and character-aware adaptive masking aim to align generated video with agent-maintained geometry, skeleton states, and reference appearances in a real-time loop.
Method
- Build a diverse dataset of aligned proxy–video pairs: joint geometric scene proxies (depth/height) and articulated character skeleton proxies, covering 750 scenes and 178 characters.
- Use an agentic framework that maintains explicit world/game state (positions, poses, health, action phases, cooldowns), updates scene geometry and skeleton motion from user inputs, and enforces rule-based combat outcomes.
- Train a causal, few-step autoregressive video generator conditioned on visual proxies and character reference images, using character-aware adaptive masking (skeleton-derived spatial masks) and causal adaptation + few-step distillation to enable real-time rollouts at 24 FPS.
Limitation
Wizard relies on a motion library, making it difficult to find compatible motions for characters with uncommon body structures. The authors also note proxy–video consistency may decline over longer rollouts.
Abstract (from arXiv)
Recent advances in video world models have enabled the generation of increasingly realistic and interactive virtual environments, opening new possibilities for player-customizable gameplay. Agentic frameworks further advance this direction by decoupling explicit game logic and controllable world representations from visual generation. However, existing approaches support only simple game settings and predefined motion patterns, generalize poorly to unseen environments and characters, and offer limited real-time interactivity. In this paper, we introduce Wizard, an agentic game world model that enables real-time, precise control over character motions while generalizing zero-shot across novel scenes, characters, and actions. To achieve this, we construct a diverse dataset spanning 750 scenes and 178 characters, incorporating skeleton and geometry information as structured representations. We leverage these representations as intermediate proxies to condition a video generation model, together with an adaptive masking mechanism that improves alignment between the conditioning signals and generated video rollouts. Furthermore, we introduce an agentic framework that explicitly maintains and updates game states, allowing players to freely configure their virtual worlds and exercise fine-grained control over character motions while preserving visual quality and gameplay consistency. By combining agentic control with structured proxy-based generation, our framework unlocks new possibilities for player-driven game creation with real-time, generalizable, and mechanically consistent world simulation.
Related papers
- PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State
- AgentGarten: Code Worlds for Evolving Agents
- ActionRoPE: action as coordinates for spatially consistent control in planar-camera game world models
- Player-1: Learning to Play from Synthetic Experience
- BoxWorld: Let World Engines Control and Video Generators Render
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
Wizard: Manipulate Any World in Real Time via Agentic Control
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.40339
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40339v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent advances in video world models have enabled the generation of increasingly realistic and interactive virtual environments, opening new possibilities for player-customizable gameplay. Agentic frameworks further advance this direction by decoupling explicit game logic and controllable world representations from visual generation. However, existing approaches support only simple game settings and predefined motion patterns, generalize poorly to unseen environments and characters, and offer limited real-time interactivity. In this paper, we introduce Wizard, an agentic game world model that enables real-time, precise control over character motions while generalizing zero-shot across novel scenes, characters, and actions. To achieve this, we construct a diverse dataset spanning 750 scenes and 178 characters, incorporating skeleton and geometry information as structured representations. We leverage these representations as intermediate proxies to condition a video generation model, together with an adaptive masking mechanism that improves alignment between the conditioning signals and generated video rollouts. Furthermore, we introduce an agentic framework that explicitly maintains and updates game states, allowing players to freely configure their virtual worlds and exercise fine-grained control over character motions while preserving visual quality and gameplay consistency. By combining agentic control with structured proxy-based generation, our framework unlocks new possibilities for player-driven game creation with real-time, generalizable, and mechanically consistent world simulation.
