Driving · 2026-11-10

Breaking survivorship bias in driving world models

Zihao Sheng, Zilin Huang, Yuhao Luo, Zhengyang Wan, Boyue Wang, Soyoung Ahn, David A. Noyce, Sikai Chen

arXiv:2611.11516PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors identify a survivorship bias in driving world models: prevailing driving datasets mostly record futures that end safely, so existing models often imagine benign outcomes even when an imminent collision would occur. They introduce a Neutral World Model (NWM) trained on routine driving plus real accident videos with a bounding-box-guided objective, and use it for reward-based action selection. The NWM more faithfully anticipates collisions and improves trajectory selection on NAVSIM and zero-shot pedestrian dummy crossing data.

Why it matters

For safety-critical autonomous driving, a world model’s reliability depends on whether it represents dangerous futures. The paper argues that current driving datasets encode survivorship bias toward safe endings, leading models to be “optimistic” about collision outcomes. By correcting this bias and using the model for imagined rollouts in decision-making, the approach aims to improve both future anticipation and action selection when multiple candidate actions must be compared.

Method

  • Introduce a neutral controllable latent video diffusion model (NWM) conditioned on initial observation, an ego trajectory, and optional surrounding agent bounding boxes; architecture builds on a pretrained Stable Video Diffusion backbone with a ControlNet adapter.
  • Train the NWM on routine driving (nuScenes) and real accident videos (MM-AU) using a two-stage procedure and a bounding-box-guided loss that increases weight on latent regions occupied by surrounding agents.
  • Use NWM as an imagination engine for reward-based action selection: for K candidate ego trajectories, generate future videos without agent controls, score them with a safety/progress/comfort reward, and select the highest-reward trajectory; evaluate on NAVSIM subset and zero-shot real-world pedestrian dummy crossing trials.
Abstract (from arXiv)

World models allow autonomous vehicles to imagine the consequences of candidate actions before execution, but their reliability depends on the futures represented in their training data. We identify a survivorship bias in driving world models: because the futures recorded in prevailing driving datasets almost always end safely, existing models often imagine benign outcomes even when a collision is imminent. We introduce a neutral world model (NWM) that learns both routine and safety-critical futures from a mixture of routine driving and real accident videos with a bounding-box-guided objective. The NWM more faithfully anticipates imminent collisions, while maintaining a low false collision rate and competitive generation quality in routine driving. We further use the model as an imagination engine for decision-making. Given multiple ego trajectories, the NWM predicts the future associated with each candidate, and a reward function evaluates these imagined outcomes before selecting an action. On challenging NAVSIM scenarios, this procedure selects the better trajectory more reliably than the baseline world model. Zero-shot experiments with real-world pedestrian dummy crossing data collected using an autonomous research vehicle further demonstrate transfer to unseen sensor observations. These results show that correcting survivorship bias enables world models to more faithfully reason about futures and support safety-critical decision-making in autonomous driving.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

Breaking survivorship bias in driving world models

Zihao Sheng, Zilin Huang, Yuhao Luo, Zhengyang Wan, Boyue Wang, Soyoung Ahn, David A. Noyce, Sikai Chen

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.11516
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11516v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

World models allow autonomous vehicles to imagine the consequences of candidate actions before execution, but their reliability depends on the futures represented in their training data. We identify a survivorship bias in driving world models: because the futures recorded in prevailing driving datasets almost always end safely, existing models often imagine benign outcomes even when a collision is imminent. We introduce a neutral world model (NWM) that learns both routine and safety-critical futures from a mixture of routine driving and real accident videos with a bounding-box-guided objective. The NWM more faithfully anticipates imminent collisions, while maintaining a low false collision rate and competitive generation quality in routine driving. We further use the model as an imagination engine for decision-making. Given multiple ego trajectories, the NWM predicts the future associated with each candidate, and a reward function evaluates these imagined outcomes before selecting an action. On challenging NAVSIM scenarios, this procedure selects the better trajectory more reliably than the baseline world model. Zero-shot experiments with real-world pedestrian dummy crossing data collected using an autonomous research vehicle further demonstrate transfer to unseen sensor observations. These results show that correcting survivorship bias enables world models to more faithfully reason about futures and support safety-critical decision-making in autonomous driving.