Video World Models · 2026-10-30

L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim, Kaeun Han, Yoonjin Yoon, Seungryong Kim

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce Latent 3D Gaussian Memory (L3G), a compact 3D memory that grows with scene extent rather than generated stream length. L3G-World alternates a memory updater (L3G-Mem) and a memory-conditioned video diffusion transformer (L3G-Gen) to generate spatially consistent worlds in real time. They report 9.02 FPS on one H100 GPU and large efficiency gains versus prior explicit 3D-memory methods.

Why it matters

Maintaining spatial consistency over long-horizon exploration in interactive, real-time video world models is challenging because temporal context and explicit 3D memories both scale poorly. The authors propose a persistent latent 3D representation that avoids dense per-frame accumulation and renders directly into the diffusion model’s latent space, aiming to enable long-horizon spatial consistency without prohibitive memory growth or latency.

Method

L3G-World alternates autoregressively between L3G-Mem (selective spawn/refine of latent 3D Gaussians) and L3G-Gen (memory-conditioned autoregressive video diffusion).
L3G-Mem selectively spawns new Gaussians only in regions marked by low opacity/high reconstruction error, and refines visible existing Gaussians; memory size grows with explored scene extent.
Each Gaussian stores a VAE latent feature so the memory can be rendered directly into L3G-Gen’s latent space without VAE re-encoding, enabling real-time conditioning.

Limitation

Latent 3D Gaussian memory is primarily designed for static scenes and does not explicitly model independently moving objects; extending it to disentangle dynamic content from static geometry is suggested for future work. Also, memory can still increase under extremely large-scale exploration, and errors may propagate through long-horizon autoregressive refinement updates.

Abstract (from arXiv)

Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to $38.6\times$ fewer 3D primitives, $3.2\times$ faster memory updates, and $359\times$ faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC

preprint