Video World Models · 2026-10-21
AEON: Unifying Video and 3D World Models
Weiqi Zhang, Wenyuan Zhang, Yu-Shen Liu, Junsheng Zhou
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AEON unifies video and 3D world models by generating explicit 4D Gaussian Splatting (4DGS) primitives from text/image/video latents. It trains a 4D Gaussian Reconstruction Model (4DRM) from monocular videos to predict 4DGS attributes and camera poses, then uses a domain adapter to translate pretrained video diffusion latents into reconstructive 4D latents via distillation. AEON supports persistent 4D environments with scene dynamics for rendering and simulation.
Why it matters
The authors argue world models need realistic appearance, dynamic evolution, 3D-consistent geometry, persistence, and real-time exploration. AEON’s approach uses 4DGS as an explicit, renderable 4D representation and bridges generative video latents with reconstructive 4D latents using an adapter and distillation, aiming to combine video realism with 3D consistency and dynamic persistence.
Method
- Train a 4D Gaussian Reconstruction Model (4DRM) from monocular posed videos using differentiable rendering; four heads predict depth, camera poses, 4D Gaussian appearance, and evolution (temporal reference/duration/velocity).
- For generation, use a pretrained video diffusion model (e.g., Wan) to produce video latents, then learn a domain adapter that maps video latent space to 4D reconstruction latent space via distillation.
- Optimize via a multi-stage pipeline: Stage 1 reconstruction training, Stage 2 end-to-end distillation from a frozen 4DRM teacher, and Stage 3 rendering refinement.
Abstract (from arXiv)
Current world models for explicit world simulation typically follow two distinct pathways: (1) video generative models (e.g., Sora), which simulate realistic appearances but fail to preserve 3D consistency; and (2) 3D generative models (e.g., Marble), which generate geometrically consistent scenes but often yield static, less realistic observations. We present AEON, a unified framework that bridges video-based and 3D-based world models and combines the benefits of both. The key insight of AEON lies in a generative model that operates within the text- or image-conditioned video latent space while producing renderable 4D graphics primitives---specifically, 4D Gaussian Splatting (4DGS)---as the explicit world representation. We train a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, serving as our initialized 4D decoder. Furthermore, we design a domain adapter that translates generative video latents into reconstructive 4D latents through a distillation process on the video VAE. AEON naturally supports video-based world modeling by rendering 4DGS at predicted camera trajectories, and constructs a persistent 4D environment with scene dynamics for realistic world simulation. Consequently, AEON empowers a diverse range of world modeling applications and achieves state-of-the-art performance on standard benchmarks. Project Page: https://weiqi-zhang.github.io/AEON
Related papers
- WorldProbe: When Video World Models Fall Apart under Long-Horizon Spatial Exploration
- L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory
- 4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling
- Orbia: Do Generated Videos Form a Consistent 3D World?
- Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
AEON: Unifying Video and 3D World Models
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28631
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Current world models for explicit world simulation typically follow two distinct pathways: (1) video generative models (e.g., Sora), which simulate realistic appearances but fail to preserve 3D consistency; and (2) 3D generative models (e.g., Marble), which generate geometrically consistent scenes but often yield static, less realistic observations. We present AEON, a unified framework that bridges video-based and 3D-based world models and combines the benefits of both. The key insight of AEON lies in a generative model that operates within the text- or image-conditioned video latent space while producing renderable 4D graphics primitives---specifically, 4D Gaussian Splatting (4DGS)---as the explicit world representation. We train a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, serving as our initialized 4D decoder. Furthermore, we design a domain adapter that translates generative video latents into reconstructive 4D latents through a distillation process on the video VAE. AEON naturally supports video-based world modeling by rendering 4DGS at predicted camera trajectories, and constructs a persistent 4D environment with scene dynamics for realistic world simulation. Consequently, AEON empowers a diverse range of world modeling applications and achieves state-of-the-art performance on standard benchmarks. Project Page: https://weiqi-zhang.github.io/AEON