Video World Models · 2026-10-15
Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors
Henry Che, Xuanchi Ren, Sanja Fidler, Shenlong Wang, Laura Leal-Taix\'e, Shengyu Huang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
LidarVerse is a real-time unified LiDAR world model that unifies generative simulation and reconstruction-grounded resimulation across tasks and sensors. A universal attention-based tokenizer maps LiDAR range sequences into a shared latent space across 32/64/128-beam sensors and supports unseen scanning patterns. A bidirectional diffusion teacher is distilled into a four-step causal student with rolling temporal KV caching and a world-aligned persistent spatial cache for long-horizon rollouts at 11 FPS on a single H100 GPU.
Why it matters
The authors target a practical LiDAR simulator that can (1) generate new worlds, (2) forecast future point clouds from a LiDAR prefix, and (3) resimulate observed scenes under new sensor trajectories—without per-scene optimization. They achieve this by switching the available conditioning evidence while keeping one model checkpoint and by using caches to maintain long-horizon temporal coherence in real time.
Method
- Use a universal LiDAR-native tokenizer: represent each sweep as a range image (range + validity, optional intensity) and encode sequences with fully attention-based (neighborhood) spatial attention and factorized temporal attention (no temporal downsampling, pt=1).
- Train a bidirectional diffusion teacher on a shared latent space with an interface that is conditioned independently on LiDAR prefix, HD-Map/actor layout, persistent spatial cache (PSC), and optionally text.
- Distill to a four-step block-autoregressive causal student with rolling temporal KV cache and a world-aligned PSC to enable persistent, temporally coherent long-horizon rollouts at real-time inference rates.
Abstract (from arXiv)
We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.
Related papers
- LO-World: Unified BEV Latent Modeling for Joint 4D Occupancy and LiDAR Forecasting
- Let LiDAR Decide Where to Look: Geometry-Measured Token Selection for Efficient Sparse Latent World Models
- Robust LIDAR Sensor-to-Sensor Domain Generalization via Accumulated-Point Scene Completion
- CRISP: Fixing Flying Pixels in Latent LiDAR Generation via Diffusion Decoding
- G$^2$WAM: World Action Models for Efficient and Geometry Grounded Image-Goal Navigation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-15 UTC
Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.19201
Explicit source versions (1)
- v1 · 2026-10-13
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.
