Video World Models · 2026-10-30
LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation
Oliver Hahn, Rodrigo Marcuzzi, Sangeetha Grama Srinivasan, Alessandro Burzio, Sherwin Bahmani, Jiahui Huang, Stefan Roth, Shengyu Huang, Qunjie Zhou, Laura Leal-Taix\'e
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
LucidLyra distills frozen 2D foundation-model predictions from a camera-controlled video diffusion model into a feed-forward decoder that outputs queryable 3D Gaussian Splatting scenes. From a single image, it predicts geometry, appearance, open-vocabulary semantic features, and class-agnostic instance features without per-scene optimization, manual 3D annotations, or captured multi-view data; it also extends to monocular video dynamics.
Why it matters
The authors target a gap between generative 3D reconstructions (renderable but unstructured) and robotic/physical-AI needs (semantics and instance boundaries for interaction). Their approach aims to produce a consistent 3D representation by aligning multi-task 2D supervision across generated views and attaching semantic/instance fields directly to Gaussian primitives in a single feed-forward pass.
Method
- Use a camera-controlled video diffusion model to generate multi-view latents from an input image along camera trajectories.
- Decode latents into 3D Gaussian Splatting primitives via an amortized, feed-forward decoder; add an input-image feature conditioning branch and two heads for reconstruction (geometry/appearance) and understanding (semantic/instance descriptors).
- Distill supervision from frozen 2D foundation models on decoded video frames, associating cross-view objects via instance tracklets and training with reconstruction plus semantic and instance contrastive losses.
Limitation
The authors state that cross-view feature jitter and mask inconsistency can occur when using 2D vision model outputs, motivating their track-consistent distillation approach; they also note that occasional grouping errors may still occur in their learned instance representations.
Abstract (from arXiv)
We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.
Related papers
- SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space
- L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory
- ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis
- SceneBloom: Agentic Indoor Environment Generation with Realistic and Diverse Layouts for Embodied Learning
- FlowVLA: Enabling World Dynamics Understanding through Pluggable 3D Flow Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.40495
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40495v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.
