Driving · 2026-10-12

Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving

Shiva Sreeram, Erfan Aasi, Tsun-Hsuan Wang, Alaa Maalouf, Daniela Rus

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors evaluate whether video-language models reliably infer ego-motion from driving videos. They find standard, visual, few-shot, and chain-of-thought prompting are unreliable and highly model/task dependent, with persistent class biases. In contrast, lightweight LoRA finetuning and simple probes on intermediate visual features recover strong motion-discriminative performance, suggesting motion cues exist internally but are not reliably used by the video-to-language pathway.

Why it matters

Driving is safety-critical and requires temporal understanding of ego-motion. The paper shows that current VidLLMs can exhibit strong class biases and fail on temporal ego-dynamics under prompting, despite motion information being decodable in intermediate representations. This cautions against treating fluent video-language outputs as dependable temporal world-model reasoning for driving without explicit temporal grounding and dynamics-aware training.

Method

Use controlled ego-motion tasks (forward/backward, left/right, acceleration/deceleration) on CARLA third-person driving videos; model answers binary questions based on different prompting strategies and an additional static reference object.
Intervene at multiple levels: standard prompting, visual prompting (green bounding box on static object), visual-text fusion few-shot, chain-of-thought prompting, and combinations; then test learning via LoRA finetuning and representation-level feature probes with lightweight classifiers.
Train/evaluate probes on intermediate visual features extracted immediately before and after each model’s visual compression/connector/merger stage to diagnose where motion information is preserved or lost.

Limitation

The authors note that they do not include single-frame or frame-shuffling controls, so high LoRA and probe accuracy cannot by itself establish that cues are genuinely temporal; they state this is left to future work.

Abstract (from arXiv)

Modern video-language models are increasingly treated as general-purpose video reasoners, yet it remains unclear whether they robustly capture temporal dynamics in safety-critical settings such as driving. We study this question through controlled ego-motion tasks that require identifying actions from driving videos. Across video-language models, we find that standard prompting, visual prompting, few-shot prompting, and chain-of-thought prompting remain unreliable and highly task- and model-dependent, with persistent class biases, although newer models show substantial task-specific few-shot gains. In contrast, lightweight LoRA finetuning and simple probes trained on intermediate visual features recover strong motion-discriminative performance, indicating that motion-relevant cues are often present inside the models but are not reliably used by the final video-to-language pathway. These results suggest that, in driving, current video-language models remain unreliable temporal reasoners and often behave more like frame-level, bias-driven recognizers than dependable temporal world models.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-12 UTC

preprint