Driving · 2026-11-13

Evaluating Video LLMs' Understanding of Corner Cases in Autonomous Driving

Gabriel Manalu, Minh Son Hoang, Hyeontaek Hwang, Jaesung Lee, Hyojin Choi, Tran Tuan Phong, Seon Kwon Kim, Saad Wazir, Daeyoung Kim

arXiv:2611.16767PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors report DashBench, a benchmark for evaluating 25 video LLMs on 1,709 ego-view dashcam clips spanning 88 scenario-level corner-case categories, balanced across collision and non-collision hazards. Using multi-prompt, open-ended QA, the authors find a capability gap: even the top model reaches 46.52% accuracy and models often miss dangerous events and hallucinate nonexistent agents/actions. Code/data are shared at cornercase-dashbench.github.io.

Why it matters

Existing driving benchmarks mainly cover routine conditions and overlook rare, safety-critical corner cases. The authors’ evaluation targets these scenarios with open-ended QA and diagnoses structured failures (corner-case blindness and road agent/action hallucination). They also test whether prompting changes or temporally local windows help, and report that gains remain well below what safety-critical applications would require, motivating new approaches.

Method

  • Introduce DashBench: 1,709 ego-view dashcam video clips across 88 corner-case categories, balanced between collision-related and non-collision hazards.
  • Evaluate 25 video LLMs using extensive prompts and open-ended question answering under a standardized multi-frame protocol (16 frames uniformly sampled per clip unless otherwise stated).
  • Analyze qualitative failure modes (corner-case blindness; road agent/action hallucination) and test prompt/window variations and hallucination mitigation techniques.

Limitation

The authors do not claim exhaustive coverage of all possible corner cases, noting that no finite collection of videos can enumerate all possible real-world road hazards.

Abstract (from arXiv)

Natural language explanations promised by modern Video Large Language Models (Video LLMs) offer autonomous driving systems a path toward detailed scene description and interpretable decision making on the road. While recent benchmarks have begun assessing these models on driving scenes, they mainly target routine traffic conditions and general comprehension tasks based on existing driving datasets and therefore fail to evaluate model performance on the rarest and most safety-critical driving events, also known as "corner cases". To provide a comprehensive evaluation of Video LLMs on precisely these scenarios, we introduce DashBench, a benchmark testing 25 Video LLMs on 1,709 carefully curated driving videos spanning not only collision-related corner cases but also rare non-collision ones. Through extensive prompts and open-ended question answering, we expose a critical capability gap: all evaluated models perform substantially below the threshold required for safety-critical driving applications, frequently misjudging or missing dangerous events on the road ahead and hallucinating non-existent agents or actions in their explanations. We believe DashBench, combining an extensive dataset with a rigorous evaluation toolkit and a detailed diagnosis of current model limitations, drives progress toward safer, more explainable autonomous driving systems. The code and data will be shared at: cornercase-dashbench.github.io

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-12 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Evaluating Video LLMs' Understanding of Corner Cases in Autonomous Driving

Gabriel Manalu, Minh Son Hoang, Hyeontaek Hwang, Jaesung Lee, Hyojin Choi, Tran Tuan Phong, Seon Kwon Kim, Saad Wazir, Daeyoung Kim

Preprint posted: 2026-11-12 · Original record ↗

arxiv
2611.16767
Explicit source versions (1)
  • v1 · 2026-11-12
  1. Preprint available

    Source date: 2026-11-12 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Natural language explanations promised by modern Video Large Language Models (Video LLMs) offer autonomous driving systems a path toward detailed scene description and interpretable decision making on the road. While recent benchmarks have begun assessing these models on driving scenes, they mainly target routine traffic conditions and general comprehension tasks based on existing driving datasets and therefore fail to evaluate model performance on the rarest and most safety-critical driving events, also known as "corner cases". To provide a comprehensive evaluation of Video LLMs on precisely these scenarios, we introduce DashBench, a benchmark testing 25 Video LLMs on 1,709 carefully curated driving videos spanning not only collision-related corner cases but also rare non-collision ones. Through extensive prompts and open-ended question answering, we expose a critical capability gap: all evaluated models perform substantially below the threshold required for safety-critical driving applications, frequently misjudging or missing dangerous events on the road ahead and hallucinating non-existent agents or actions in their explanations. We believe DashBench, combining an extensive dataset with a rigorous evaluation toolkit and a detailed diagnosis of current model limitations, drives progress toward safer, more explainable autonomous driving systems. The code and data will be shared at: cornercase-dashbench.github.io