Driving · 2026-11-02

Let LiDAR Decide Where to Look: Geometry-Measured Token Selection for Efficient Sparse Latent World Models

Zeping Zhang, Zhuoya Zhao, Robert Lagani\`ere

arXiv:2610.40935PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors present Geometry-Aware Token Selection (GATS), which uses LiDAR and radar point projections to select a small fraction of image tokens before the vision backbone runs in a sparse latent world model. At 25% token budget, GATS improves nuScenes 3-sec L2 displacement error and NAVSIM EPDMS while reducing end-to-end FLOPs; at 10% budget it still beats the dense baseline. It also reports safety-critical coverage and collision-rate improvements on intent-critical scenarios.

Why it matters

Dense vision token sets can dilute the downstream planner’s attention and dominate compute. By selecting tokens using measured 3D geometry before feature extraction, the approach reduces irrelevant attention while cutting visual backbone FLOPs, and still trains a self-supervised latent world model without box annotations.

Method

  • GATS-WM selects top-K image tokens using LiDAR and radar geometry (obstacle/road/visual-reserve streams) and keeps a fixed fraction per frame before the vision backbone runs.
  • Each kept token gets a 3D anchor from its LiDAR points (or a virtual anchor when no LiDAR points fall in the patch) and uses 3D-anchored metadata for cross-camera association and motion prediction.
  • The self-supervised latent world model predicts future visual features and 3D displacements (JEPA-style feature prediction + anchor/match losses) without box annotations, followed by a planning head with collision-cost scoring.

Limitation

The method requires LiDAR, which limits applicability to camera-only vehicles. The authors also note patch resolution (14×14 pixels) is too coarse for fine-grained cues such as turn signals.

Abstract (from arXiv)

Vision-centric autonomous driving models encode thousands of image tokens per frame, yet most depict sky, buildings, or empty road---content irrelevant to planning. Worse, these tokens dilute the planner's attention: we show that randomly discarding 75% of tokens already outperforms the dense model, because the planner can concentrate on the remaining driving-critical content. Building on this finding, we present Geometry-Aware Token Selection (GATS), which uses LiDAR and radar point projections to select a small fraction of image tokens before the vision backbone runs. Because the selector operates on measured 3D geometry rather than visual features, the backbone encodes only the selected patches, cutting end-to-end FLOPs by up to $18\times$. Each kept token receives a 3D anchor from its LiDAR point and is fed to a self-supervised latent world model that predicts future features and 3D displacements without box annotations. At 25% budget, GATS achieves a 3-sec L2 displacement error of 0.520 m on nuScenes and an EPDMS of 88.9 on NAVSIM---surpassing the dense baseline (0.573 m / 85.9) at $6.6\times$ fewer total FLOPs. At an extreme 10% budget (24 GFLOPs, $18\times$ reduction), it still beats the dense baseline on both nuScenes (0.537 m vs. 0.573 m L2) and NAVSIM (87.9 vs. 85.9 EPDMS). On 739 intent-critical scenarios, GATS covers 95.8% of hazardous vehicles and achieves a 0.09% collision rate---73.5% lower than the dense model (0.34%).

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-02 · Indexed by Pub-AI: 2026-11-02 UTC

preprint

Let LiDAR Decide Where to Look: Geometry-Measured Token Selection for Efficient Sparse Latent World Models

Zeping Zhang, Zhuoya Zhao, Robert Laganière

First announced by arXiv: 2026-11-02 · Original record ↗

arxiv
2610.40935
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-02 · First detected: 2026-11-02 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40935v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Vision-centric autonomous driving models encode thousands of image tokens per frame, yet most depict sky, buildings, or empty road---content irrelevant to planning. Worse, these tokens dilute the planner's attention: we show that randomly discarding 75% of tokens already outperforms the dense model, because the planner can concentrate on the remaining driving-critical content. Building on this finding, we present Geometry-Aware Token Selection (GATS), which uses LiDAR and radar point projections to select a small fraction of image tokens before the vision backbone runs. Because the selector operates on measured 3D geometry rather than visual features, the backbone encodes only the selected patches, cutting end-to-end FLOPs by up to $18\times$. Each kept token receives a 3D anchor from its LiDAR point and is fed to a self-supervised latent world model that predicts future features and 3D displacements without box annotations. At 25% budget, GATS achieves a 3-sec L2 displacement error of 0.520 m on nuScenes and an EPDMS of 88.9 on NAVSIM---surpassing the dense baseline (0.573 m / 85.9) at $6.6\times$ fewer total FLOPs. At an extreme 10% budget (24 GFLOPs, $18\times$ reduction), it still beats the dense baseline on both nuScenes (0.537 m vs. 0.573 m L2) and NAVSIM (87.9 vs. 85.9 EPDMS). On 739 intent-critical scenarios, GATS covers 95.8% of hazardous vehicles and achieves a 0.09% collision rate---73.5% lower than the dense model (0.34%).