Driving · 2026-10-20

Cartender: Verifiable Language-Conditioned Driving by Learning the Drives Not Taken

Zhiwen Chen, Benjin Zhu, Yan Xie, Jun Ma

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Cartender learns verifiable language-conditioned driving from logged data where each scene is recorded once. It uses Counterfactual Branching to construct matched/mismatched instruction-action pairs from real recordings (no simulator), then gates language conditioning with a learned scene-language compatibility score and ranks candidates with a reward-trained language residual. Evaluation on WOD-E2E uses behavior under instruction swapping.

Why it matters

The authors report that instruction-following scores on matched instruction–trajectory pairs do not establish that policies depend on language, because logs only include the executed action and multiple admissible behaviors fit the same data. Cartender targets this confound by creating same-scene counterfactual supervision and evaluating instruction dependence via behavior changes under swapped instructions.

Method

Counterfactual Branching: constructs matched and mismatched instruction-action pairs from real recordings without a simulator; feasible instructions get teacher trajectories, infeasible instructions get negative applicability labels, via rule-based admission after reconstruction checks.
Compatibility-gated generation: trains a continuous scene–language compatibility score to regulate classifier-free guidance strength during conditional trajectory generation (language strength interpolates with compatibility).
Reward-trained reranking: initializes candidate ranking from teacher proximity, then refines with a language residual trained via GRPO using a frozen key-event evaluator reward.

Limitation

Rule-based admission and RFS do not certify collision freedom or instruction fulfillment, and the gate does not eliminate unsupported execution (e.g., infeasible instructions can still be executed).

Abstract (from arXiv)

Every driving scene is logged once, but driving is one-to-many: the scene determines which behaviors are feasible, while an instruction selects among them. A log records only the action taken, leaving no direct supervision for alternative feasible actions or instructions the scene cannot support. Consequently, a policy that ignores language and one that follows it without regard to scene constraints can fit the same data. We present Cartender, which learns from the drives not taken. Counterfactual Branching constructs matched and mismatched instruction-action pairs from real recordings without a simulator: a vision-language model proposes alternative futures and their selecting instructions, while unsupported instructions provide negative labels. A learned scene-language compatibility score regulates language conditioning in trajectory generation, and a reward-trained language residual re-ranks generated candidates over a scene-based score. The branches are used for training; instruction following is evaluated by swapping instructions under identical recorded inputs and measuring behavior rather than agreement with generated targets. On WOD-E2E, the best of four proposals achieves an RFS of 8.642 across 3,603 feasible instruction conditions. Across 8,974 admitted proposals, mean RFS is 8.203 versus 7.906 for matched logged trajectories, and 67% score higher than their logged counterpart. Compared with matched-only imitation using the same backbone, Cartender improves same-scene paired instruction success from 23.67% to 57.00% and RFS from 6.32 to 7.10; its gate reduces infeasible-instruction execution from 53.36% to 14.18%.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC

preprint