Video World Models · 2026-10-30

LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Oliver Hahn, Rodrigo Marcuzzi, Sangeetha Grama Srinivasan, Alessandro Burzio, Sherwin Bahmani, Jiahui Huang, Stefan Roth, Shengyu Huang, Qunjie Zhou, Laura Leal-Taix\'e

arXiv:2610.40495PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

LucidLyra distills frozen 2D foundation-model predictions from a camera-controlled video diffusion model into a feed-forward decoder that outputs queryable 3D Gaussian Splatting scenes. From a single image, it predicts geometry, appearance, open-vocabulary semantic features, and class-agnostic instance features without per-scene optimization, manual 3D annotations, or captured multi-view data; it also extends to monocular video dynamics.

Why it matters

The authors target a gap between generative 3D reconstructions (renderable but unstructured) and robotic/physical-AI needs (semantics and instance boundaries for interaction). Their approach aims to produce a consistent 3D representation by aligning multi-task 2D supervision across generated views and attaching semantic/instance fields directly to Gaussian primitives in a single feed-forward pass.

Method

  • Use a camera-controlled video diffusion model to generate multi-view latents from an input image along camera trajectories.
  • Decode latents into 3D Gaussian Splatting primitives via an amortized, feed-forward decoder; add an input-image feature conditioning branch and two heads for reconstruction (geometry/appearance) and understanding (semantic/instance descriptors).
  • Distill supervision from frozen 2D foundation models on decoded video frames, associating cross-view objects via instance tracklets and training with reconstruction plus semantic and instance contrastive losses.

Limitation

The authors state that cross-view feature jitter and mask inconsistency can occur when using 2D vision model outputs, motivating their track-consistent distillation approach; they also note that occasional grouping errors may still occur in their learned instance representations.

Abstract (from arXiv)

We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC

preprint

LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Oliver Hahn, Rodrigo Marcuzzi, Sangeetha Grama Srinivasan, Alessandro Burzio, Sherwin Bahmani, Jiahui Huang, Stefan Roth, Shengyu Huang, Qunjie Zhou, Laura Leal-Taixé

Preprint posted: 2026-10-29 · Original record ↗

arxiv
2610.40495
Explicit source versions (1)
  • v1 · 2026-10-29
  1. Preprint available

    Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.