Video World Models · 2026-10-14
Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering
Hanxin Zhu, Jianqi Ma, Cong Wang, Zhihao Hu, Junyao Gao, Tianyu He, Junyan Lin, Jiayin Xiao, Jiateng Shou, Sen Liang, Weidong Zhang, Zhibo Chen, Jun Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Demiurge is an autoregressive video rerendering pipeline for creating coherent, traversable 4D worlds. It maintains an evolving dynamic 4D point-cloud memory shared across target camera trajectories, using memory-conditioned appearance diffusion, memory-consistent depth completion, memory-aware dual-stream warping, and optional geometry-aware refinement. The authors report state-of-the-art performance and show coherent spatiotemporal lattices, large-parallax rerendering (up to 180° rotations), and geometry-faithful outputs usable for 3D Gaussian Splatting.
Why it matters
The authors argue that existing camera-conditioned rerendering methods synthesize each trajectory independently, causing cross-trajectory inconsistencies and preventing globally consistent 4D world construction. By sharing a dynamic 4D point-cloud memory across autoregressive steps, Demiurge aims to enforce cross-trajectory consistency for a coherent and traversable 4D representation.
Method
- Maintain an evolving 4D point-cloud memory that serves as persistent world state; at each autoregressive step, render the memory along a target trajectory to condition novel-view generation, then back-project recovered RGB-D to update the memory.
- Use a memory-guided appearance video diffusion model with a lightweight noise-monitor to suppress unreliable rendered guidance before denoising.
- Recover temporally coherent dense depth via memory-consistent diffusion depth completion anchored to cached geometry, then update memory using memory-aware dual-stream warping (appearance vs geometry streams) and optional geometry-aware video refinement.
Limitation
The primary limitation of Demiurge is its inference latency: the current pipeline comprises several sequentially coupled modules (appearance generation, video depth completion, optional geometry-aware refinement, and 4D memory update). They must exchange and synchronize intermediate RGB, depth, and geometric representations before the next autoregressive step, preventing straightforward parallel execution; the full pipeline requires approximately five minutes to generate one target-trajectory video, limiting interactive use.
Abstract (from arXiv)
Recent advances in video diffusion enable novel-view video synthesis from a single input video, supporting world modeling, embodied intelligence, and robotic simulation. However, existing methods process camera trajectories independently, lacking the cross-trajectory consistency needed for coherent, navigable 4D worlds. We introduce Demiurge, an autoregressive video rerendering pipeline built on a dynamic 4D point-cloud memory cache. It promotes cross-trajectory consistency through four components: (1) an appearance video diffusion model that synthesizes photorealistic novel-view videos conditioned on the 4D memory; (2) a memory-consistent video depth completion model that aligns known-region depths with cached geometry and completes missing regions, mitigating scale ambiguity and discontinuities during memory updates; (3) memory-aware dual-stream warping that provides high-fidelity visual guidance and occlusion cues for novel-view synthesis; and (4) geometry-aware video refinement that enhances perceptual quality. Extensive experiments demonstrate state-of-the-art performance and three key capabilities: coherent spatiotemporal lattice construction over dense multi-view trajectories; large-parallax video rerendering under challenging camera motions, including 180-degree rotations with no input-view overlap; and geometry-faithful outputs that can be directly optimized into high-quality 3D Gaussian Splatting representations with minimal artifacts. Project page: https://hanxinzhu-lab.github.io/demiurge_project_page/.
Related papers
- 4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling
- Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution
- Persistent Planar Memory for Video World Models
- WorldProbe: When Video World Models Fall Apart under Long-Horizon Spatial Exploration
- Learning the 4D World through Point Trajectory Forecasting
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-10 · Indexed by Pub-AI: 2026-10-14 UTC
Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering
Preprint posted: 2026-10-10 · Original record ↗
- arxiv
- 2610.14685
Explicit source versions (1)
- v1 · 2026-10-10
- Preprint available
Source date: 2026-10-10 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent advances in video diffusion enable novel-view video synthesis from a single input video, supporting world modeling, embodied intelligence, and robotic simulation. However, existing methods process camera trajectories independently, lacking the cross-trajectory consistency needed for coherent, navigable 4D worlds. We introduce Demiurge, an autoregressive video rerendering pipeline built on a dynamic 4D point-cloud memory cache. It promotes cross-trajectory consistency through four components: (1) an appearance video diffusion model that synthesizes photorealistic novel-view videos conditioned on the 4D memory; (2) a memory-consistent video depth completion model that aligns known-region depths with cached geometry and completes missing regions, mitigating scale ambiguity and discontinuities during memory updates; (3) memory-aware dual-stream warping that provides high-fidelity visual guidance and occlusion cues for novel-view synthesis; and (4) geometry-aware video refinement that enhances perceptual quality. Extensive experiments demonstrate state-of-the-art performance and three key capabilities: coherent spatiotemporal lattice construction over dense multi-view trajectories; large-parallax video rerendering under challenging camera motions, including 180-degree rotations with no input-view overlap; and geometry-faithful outputs that can be directly optimized into high-quality 3D Gaussian Splatting representations with minimal artifacts. Project page: https://hanxinzhu-lab.github.io/demiurge_project_page/.