Video World Models · 2026-10-15

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang, Yi Wu, Linqing Zhao, Haibin Yan

arXiv:2610.19708 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SceneAE represents a scene as a sparse 3D-anchored latent point cloud where each latent has explicit 3D position plus implicit content. The authors decode RGB-D from projected latents for novel camera views without regenerating a video, and generate scenes with a two-stage “geometry-content decoupled” approach: first generate target 3D anchor geometry from camera trajectories, then synthesize content conditioned on it.

Why it matters

The paper targets camera-queryable scene generation/reconstruction by using an explicit 3D latent space. The authors claim their 3D anchoring enables fast novel-view RGB-D decoding from the same persistent representation, and improves camera controllability via explicit control over generated scene geometry followed by content synthesis.

Method

SceneAE uses sparse, unstructured 3D-anchored latents: R={P,Z} with explicit 3D coordinates p_i and implicit content features z_i, forming a latent point cloud.
For reconstruction, the persistent 3D latents are projected onto a target camera’s image plane and decoded into RGB-D via a splat-based initialization plus a learned decoder (no per-trajectory video regeneration).
For generation, a geometry-content decoupled DiT first predicts future 3D anchor layout from reference latents and target camera trajectory (Point Diffusion), then generates content features conditioned on the predicted 3D geometry (Content DiT).

Limitation

The authors state their representation assumes scene content can be anchored to persistent locations in a shared 3D coordinate system, which is restrictive for moving or deformable objects; they also note that they do not explicitly model object motion or temporal evolution and extend the representation with time-dependent anchors/motion fields or static-dynamic decomposition is necessary.

Abstract (from arXiv)

Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (\textbf{SceneAE}). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC

preprint