Video World Models · 2026-10-15

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Henry Che, Xuanchi Ren, Sanja Fidler, Shenlong Wang, Laura Leal-Taix\'e, Shengyu Huang

arXiv:2610.19201PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

LidarVerse is a real-time unified LiDAR world model that unifies generative simulation and reconstruction-grounded resimulation across tasks and sensors. A universal attention-based tokenizer maps LiDAR range sequences into a shared latent space across 32/64/128-beam sensors and supports unseen scanning patterns. A bidirectional diffusion teacher is distilled into a four-step causal student with rolling temporal KV caching and a world-aligned persistent spatial cache for long-horizon rollouts at 11 FPS on a single H100 GPU.

Why it matters

The authors target a practical LiDAR simulator that can (1) generate new worlds, (2) forecast future point clouds from a LiDAR prefix, and (3) resimulate observed scenes under new sensor trajectories—without per-scene optimization. They achieve this by switching the available conditioning evidence while keeping one model checkpoint and by using caches to maintain long-horizon temporal coherence in real time.

Method

  • Use a universal LiDAR-native tokenizer: represent each sweep as a range image (range + validity, optional intensity) and encode sequences with fully attention-based (neighborhood) spatial attention and factorized temporal attention (no temporal downsampling, pt=1).
  • Train a bidirectional diffusion teacher on a shared latent space with an interface that is conditioned independently on LiDAR prefix, HD-Map/actor layout, persistent spatial cache (PSC), and optionally text.
  • Distill to a four-step block-autoregressive causal student with rolling temporal KV cache and a world-aligned PSC to enable persistent, temporally coherent long-horizon rollouts at real-time inference rates.
Abstract (from arXiv)

We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-15 UTC

preprint

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Henry Che, Xuanchi Ren, Sanja Fidler, Shenlong Wang, Laura Leal-Taixé, Shengyu Huang

Preprint posted: 2026-10-13 · Original record ↗

arxiv
2610.19201
Explicit source versions (1)
  • v1 · 2026-10-13
  1. Preprint available

    Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.