Driving · 2026-10-14
Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, Sining Ang, Yuguang Yang, Jueyu Chen, Xiangyu Li, Qianshan Wei, Jieyuan Pei, Yuxin Wang, Zhiwen Chen, Jiaqi Li, Wenxuan Song, Yanyuan Qiao, Benjin Zhu, Haoang Li, Yan Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Language2Drive tests whether vision-language-action driving models follow natural-language instructions in autonomous driving by using Paired Counterfactual Units (PCUs) that keep the driving scene fixed while changing only the instruction. Across environments and multiple models, the authors report a large gap between individual instruction completion and paired counterfactual success, with 93.2% of successful rollouts lacking a successful opposite-instruction counterpart.
Why it matters
The authors argue that conventional closed-loop driving benchmarks can overstate language-following because isolated route success or driving scores do not confirm that changing the instruction reliably changes which feasible behavior the vehicle executes. Language2Drive’s paired counterfactual evaluation targets this mismatch, showing that many driving-capable models have weak instruction control under matched alternatives.
Method
- Paired Counterfactual Units (PCUs) evaluate two mutually exclusive but feasible instructions while keeping map, ego/actor states, history, non-language inputs, and simulation seed fixed.
- Success is machine-checkable via strict instruction completion for each request and Counterfactual Paired Success (CFPS) requiring both opposed alternatives in a PCU to succeed.
- The benchmark covers four language-control scopes (atomic actions, persistent behavior, scene-grounded navigation, long-horizon programs) across CARLA, reconstructed real-world scenes (RealSim), and open-loop evaluations.
Limitation
The authors report that no evaluated CARLA system completes both alternatives under all three meaning-preserving expressions, and that paired success can be sparse, but they do not state a specific limitation of their benchmark in the provided text beyond the evaluation shortcomings it exposes.
Abstract (from arXiv)
Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.
Related papers
- SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
- Cartender: Verifiable Language-Conditioned Driving by Learning the Drives Not Taken
- doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Driving
- YOUDrive: Driving Style as a Composable Direction for End-to-End Autonomous Driving
- Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-14 UTC
Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
Preprint posted: 2026-10-12 · Original record ↗
- arxiv
- 2610.16634
Explicit source versions (1)
- v1 · 2026-10-12
- Preprint available
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.