Driving · 2026-11-05
Jev with Foresight: Safe Language-Guided Autonomous Driving via a Language World Model
Yansong Qu, Jiancong Chen, Samuel Labi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Jev-AD combines Jev (a typed decision model) with a language world model that predicts, for each candidate action, how nearby vehicles move over the next 2 s. Code converts predictions into foreseen consequences (collision/Danger, TTC, final lane/speed), which Jev uses to choose safe actions that match passenger intent. In KoMA highway merging, foresight lowers collisions and achieves 93% intent; on replayed NGSIM traffic it also lowers collisions and foresees 93% dangerous actions.
Why it matters
The authors address two issues with LLM driving agents: slow decision-making from text generation and unsafe implicit consequence inference. Jev-AD moves driving decisions into a fast typed interface (single-call probability outputs) and uses a language world model to provide action-level foresight in short words computed by code, enabling safer closed-loop action selection.
Method
- Use Jev (typed decision model) to ground free-form commands via four typed questions (unsafe/off/dynamic intensity/maneuver) and to select among highway-env meta-actions each step based on foreseen consequences in words.
- Train a language world model that reads scene history as text and, for each candidate ego action, predicts nearby vehicles’ accelerations/steering and the other AV’s action over the next 2 s (integrated with kinematic bicycle model).
- Compute foresight via code from predicted trajectories: collision, minimum time to collision (TTC), final lane/speed, and whether merge occurred; label Danger when TTC < 1.5 s; Jev forbids options foreseen as Collision or Danger when a safe one exists.
Limitation
The NGSIM study replays recorded drivers, who react to the AV only along their recorded lanes, and no test was run on a vehicle. The world model was trained separately for each setting and not tested across settings. Jev is proprietary via an API, and evaluation uses ground-truth states, simulator low-level control, five meta-actions with 2 s foresight, and only merge/roundabout/freeway scenes.
Abstract (from arXiv)
Large language models understand passengers' driving commands, but as driving agents they reason in generated text for seconds and must infer each action's consequences implicitly from the scene. We present Jev-AD, which pairs Jev, a typed decision model that answers predefined questions with probabilities, not generated text, with a language world model that predicts for every candidate action how nearby vehicles move over the next two seconds. Code turns these predictions into foreseen consequences, which Jev compares instead of inferring them from coordinates. In simulated highway merging (500 car-episodes), this foresight lowers the same decision model's collision rate from 26.0% to 2.4% at 0.25 seconds per decision, and Jev-AD achieves the passenger's intent in 93% of free-form command trials. In 500 replayed real-world NGSIM episodes, foresight lowers the collision rate from 48.2% to 2.8%, and the world model, learned from this traffic, foresees 93% of dangerous actions. Project page: https://ys-qu.github.io/jev-ad-website.
Related papers
- SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
- doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Driving
- Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
- AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving
- Breaking survivorship bias in driving world models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-05 · Indexed by Pub-AI: 2026-11-05 UTC
Jev with Foresight: Safe Language-Guided Autonomous Driving via a Language World Model
First announced by arXiv: 2026-11-05 · Original record ↗
- arxiv
- 2611.05448
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-05 · First detected: 2026-11-05 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.05448v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Large language models understand passengers' driving commands, but as driving agents they reason in generated text for seconds and must infer each action's consequences implicitly from the scene. We present Jev-AD, which pairs Jev, a typed decision model that answers predefined questions with probabilities, not generated text, with a language world model that predicts for every candidate action how nearby vehicles move over the next two seconds. Code turns these predictions into foreseen consequences, which Jev compares instead of inferring them from coordinates. In simulated highway merging (500 car-episodes), this foresight lowers the same decision model's collision rate from 26.0% to 2.4% at 0.25 seconds per decision, and Jev-AD achieves the passenger's intent in 93% of free-form command trials. In 500 replayed real-world NGSIM episodes, foresight lowers the collision rate from 48.2% to 2.8%, and the world model, learned from this traffic, foresees 93% of dangerous actions. Project page: https://ys-qu.github.io/jev-ad-website.
