Video World Models · 2026-11-03

AgiWorld: An Agentic Video World Model for Building Persistent Worlds

Shaoheng Fang, Yiqun Mei, Hanwen Jiang, Xin Sun, Haitian Zheng, Hao Tan, Kalyan Sunkavalli, Georgios Pavlakos, Qixing Huang, Yiwei Hu

arXiv:2611.02341PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AgiWorld is an agentic video world model that builds and explores a persistent 3D world from a single image and a free-form instruction. AgiWorld-Agent plans a metric camera trajectory and structured prompts via a high-level world state. AgiWorld-WM generates video chunk-by-chunk with spatial memory (point cloud + geometry) in a closed loop, using occlusion-aware history selection and reliability-based memory maintenance for long rollouts.

Why it matters

The authors address persistence and high-level intent: the system autonomously designs the world and a coherent journey, then keeps generated frames and world state consistent as exploration proceeds. Persistence is obtained from the closed loop between an agent that updates a world state and a world model that stores generated content with geometry and updates memory for further planning.

Method

  • AgiWorld-Agent (multimodal language model) maintains a persistent world state (confirmed vs imagined layout elements and journey stages), then directs AgiWorld-WM by producing metric camera trajectories and structured text prompts; it reviews generated chunks and regenerates failed ones.
  • AgiWorld-WM generates videos autoregressively in 5-second chunks with metric camera conditioning (camera adapter + relative positional encoding PRoPE), estimates depth, and stores frames with geometry in a spatial memory (point cloud history).
  • To maintain long-rollout consistency, the authors propose occlusion-aware history selection (retrieve only history visible from the new camera) and reliability-based memory maintenance (prefer reliable geometry and earlier observations).
Abstract (from arXiv)

Recent interactive video world models generate visually compelling videos from a single image, following user-provided camera motion and text prompts. However, such low-level control demands substantial user effort to realize a high-level intent, and the generated world struggles to remain persistent as the exploration proceeds. In this work, we present AgiWorld, an agentic video world model that explores and builds a persistent world from a single image and a free-form instruction. For high-level planning, our agent AgiWorld-Agent designs the world and the journey through it in a world state and directs the exploration. For low-level realization, our video world model AgiWorld-WM generates the video chunk by chunk and stores it with its geometry in a spatial memory, from which the agent updates its world state. This closed loop keeps the world faithful to the user's intent and keeps the agent's world state consistent with the generated content throughout the exploration. We co-design the two levels around the shared spatial memory and an interface of metric camera trajectories and structured prompts. In AgiWorld-WM, to handle occlusion and accumulated geometric errors in the spatial memory over long rollouts, we propose occlusion-aware history selection, which retrieves past views by surface visibility, and reliability-based memory maintenance, which prefers reliable geometry and earlier observations. AgiWorld-WM achieves state-of-the-art overall performance on world revisiting and exploration benchmarks, and qualitative results and ablations show that AgiWorld builds persistent worlds from free-form instructions over minute-long journeys. Persistence thus arises not from the video model alone but from its closed loop with the agent.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-02 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

AgiWorld: An Agentic Video World Model for Building Persistent Worlds

Shaoheng Fang, Yiqun Mei, Hanwen Jiang, Xin Sun, Haitian Zheng, Hao Tan, Kalyan Sunkavalli, Georgios Pavlakos, Qixing Huang, Yiwei Hu

Preprint posted: 2026-11-02 · Original record ↗

arxiv
2611.02341
Explicit source versions (1)
  • v1 · 2026-11-02
  1. Preprint available

    Source date: 2026-11-02 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Recent interactive video world models generate visually compelling videos from a single image, following user-provided camera motion and text prompts. However, such low-level control demands substantial user effort to realize a high-level intent, and the generated world struggles to remain persistent as the exploration proceeds. In this work, we present AgiWorld, an agentic video world model that explores and builds a persistent world from a single image and a free-form instruction. For high-level planning, our agent AgiWorld-Agent designs the world and the journey through it in a world state and directs the exploration. For low-level realization, our video world model AgiWorld-WM generates the video chunk by chunk and stores it with its geometry in a spatial memory, from which the agent updates its world state. This closed loop keeps the world faithful to the user's intent and keeps the agent's world state consistent with the generated content throughout the exploration. We co-design the two levels around the shared spatial memory and an interface of metric camera trajectories and structured prompts. In AgiWorld-WM, to handle occlusion and accumulated geometric errors in the spatial memory over long rollouts, we propose occlusion-aware history selection, which retrieves past views by surface visibility, and reliability-based memory maintenance, which prefers reliable geometry and earlier observations. AgiWorld-WM achieves state-of-the-art overall performance on world revisiting and exploration benchmarks, and qualitative results and ablations show that AgiWorld builds persistent worlds from free-form instructions over minute-long journeys. Persistence thus arises not from the video model alone but from its closed loop with the agent.