Driving · 2026-10-01
DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?
Aditya Ramabadran, Simon Mahns, Tobias Gessler
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DrivingBench is a benchmark intended to test general-purpose vision-language models as closed-loop drivers of a real 2022 Toyota Corolla. Models view camera frames and directly command steering and velocity via an MCP tool interface; latency is part of the task because the car continues while the model thinks. GPT-6 Astra is the only model to finish, reaching the finish zone on its second attempt (100% progress).
Why it matters
The authors report (to their knowledge) the first benchmark where general-purpose vision-language models must drive a real car in the real world and in closed loop, with the model itself as the driver. This exposes failures across perception, planning, and latency, and they release an open-source evaluation harness plus prompts, course map, and traces (video/telemetry) for reproducibility. They also analyze refusal conditions and how tool output format/task framing affects whether models drive or refuse.
Method
- Real-world closed-loop driving benchmark: a 2022 Toyota Corolla retrofitted with a comma four running modified openpilot; models see camera frames and directly command steering/speed via three MCP tools (observe, set_motion, stop_now).
- Up to three attempts per model in one continuous chat with fixed reflection/continuation prompts; the car can keep executing a running command while the model thinks, making inference latency part of the task.
- Primary metric is course progress: furthest point along a 127 m centerline reached while within 4 m, reported as a share of 127 m up to the finish zone (100% means finish).
Limitation
Progress is imperfect and may not truly capture the “percent” of driving ability each model shows; the authors state they use it as a viable choice to encapsulate benchmark results into one number.
Abstract (from arXiv)
Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present DrivingBench, to our knowledge the first benchmark where general-purpose vision-language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and velocity around a parking lot cone course at low speeds. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We also detail the design principles behind our action interface, and show how the tool output format and the framing of the task combined to determine whether models would drive at all or refuse. We release our harness, prompts, course map, and traces with video and telemetry for reproducibility.
Related papers
- Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
- Evaluating Video LLMs' Understanding of Corner Cases in Autonomous Driving
- DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving
- Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving
- DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC
DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2609.38948
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present DrivingBench, to our knowledge the first benchmark where general-purpose vision-language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and velocity around a parking lot cone course at low speeds. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We also detail the design principles behind our action interface, and show how the tool output format and the framing of the task combined to determine whether models would drive at all or refuse. We release our harness, prompts, course map, and traces with video and telemetry for reproducibility.
