Driving · 2026-11-05

Jev with Foresight: Safe Language-Guided Autonomous Driving via a Language World Model

Yansong Qu, Jiancong Chen, Samuel Labi

arXiv:2611.05448PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Jev-AD combines Jev (a typed decision model) with a language world model that predicts, for each candidate action, how nearby vehicles move over the next 2 s. Code converts predictions into foreseen consequences (collision/Danger, TTC, final lane/speed), which Jev uses to choose safe actions that match passenger intent. In KoMA highway merging, foresight lowers collisions and achieves 93% intent; on replayed NGSIM traffic it also lowers collisions and foresees 93% dangerous actions.

Why it matters

The authors address two issues with LLM driving agents: slow decision-making from text generation and unsafe implicit consequence inference. Jev-AD moves driving decisions into a fast typed interface (single-call probability outputs) and uses a language world model to provide action-level foresight in short words computed by code, enabling safer closed-loop action selection.

Method

  • Use Jev (typed decision model) to ground free-form commands via four typed questions (unsafe/off/dynamic intensity/maneuver) and to select among highway-env meta-actions each step based on foreseen consequences in words.
  • Train a language world model that reads scene history as text and, for each candidate ego action, predicts nearby vehicles’ accelerations/steering and the other AV’s action over the next 2 s (integrated with kinematic bicycle model).
  • Compute foresight via code from predicted trajectories: collision, minimum time to collision (TTC), final lane/speed, and whether merge occurred; label Danger when TTC < 1.5 s; Jev forbids options foreseen as Collision or Danger when a safe one exists.

Limitation

The NGSIM study replays recorded drivers, who react to the AV only along their recorded lanes, and no test was run on a vehicle. The world model was trained separately for each setting and not tested across settings. Jev is proprietary via an API, and evaluation uses ground-truth states, simulator low-level control, five meta-actions with 2 s foresight, and only merge/roundabout/freeway scenes.

Abstract (from arXiv)

Large language models understand passengers' driving commands, but as driving agents they reason in generated text for seconds and must infer each action's consequences implicitly from the scene. We present Jev-AD, which pairs Jev, a typed decision model that answers predefined questions with probabilities, not generated text, with a language world model that predicts for every candidate action how nearby vehicles move over the next two seconds. Code turns these predictions into foreseen consequences, which Jev compares instead of inferring them from coordinates. In simulated highway merging (500 car-episodes), this foresight lowers the same decision model's collision rate from 26.0% to 2.4% at 0.25 seconds per decision, and Jev-AD achieves the passenger's intent in 93% of free-form command trials. In 500 replayed real-world NGSIM episodes, foresight lowers the collision rate from 48.2% to 2.8%, and the world model, learned from this traffic, foresees 93% of dangerous actions. Project page: https://ys-qu.github.io/jev-ad-website.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-05 · Indexed by Pub-AI: 2026-11-05 UTC

preprint

Jev with Foresight: Safe Language-Guided Autonomous Driving via a Language World Model

Yansong Qu, Jiancong Chen, Samuel Labi

First announced by arXiv: 2026-11-05 · Original record ↗

arxiv
2611.05448
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-05 · First detected: 2026-11-05 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.05448v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Large language models understand passengers' driving commands, but as driving agents they reason in generated text for seconds and must infer each action's consequences implicitly from the scene. We present Jev-AD, which pairs Jev, a typed decision model that answers predefined questions with probabilities, not generated text, with a language world model that predicts for every candidate action how nearby vehicles move over the next two seconds. Code turns these predictions into foreseen consequences, which Jev compares instead of inferring them from coordinates. In simulated highway merging (500 car-episodes), this foresight lowers the same decision model's collision rate from 26.0% to 2.4% at 0.25 seconds per decision, and Jev-AD achieves the passenger's intent in 93% of free-form command trials. In 500 replayed real-world NGSIM episodes, foresight lowers the collision rate from 48.2% to 2.8%, and the world model, learned from this traffic, foresees 93% of dangerous actions. Project page: https://ys-qu.github.io/jev-ad-website.