Driving · 2026-11-13

Driving World Models Move the Camera, Not the Scene

Yuqiao Feng, Xiaoyu Ao

arXiv:2611.15260PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors report that plan-conditioned driving world models typically move the camera but barely move the scene: their scene response to speed or lateral plans reaches at most about a third of real-video response (and at most a quarter for a 0.54 m lateral shift). The trajectory ADE score can’t distinguish executing vs ignoring speed commands due to scale alignment. They propose ReTime-Compose, which delivers the plan as geometry without training.

Why it matters

Driving world models are used to compare candidate plans, so forecasts must change with the plan as the real scene would. The paper argues that common control metrics mainly verify camera-following, not scene changes under unrecorded (counterfactual) plans. The authors’ scene-response measurement and counterfactual references expose this evaluation gap and motivate a training-free repair that makes generated forecasts use the plan via transport/geometry.

Method

  • Define scene response α: the share of true LiDAR depth change between two plans (e.g., half vs logged speed) reproduced by the forecast; evaluate against retimed logs (speed) and stereo-shifted rigs (lateral).
  • Intervene inside released generators to localize failure to a plan–geometry channel mismatch: context frames set geometry; plan tokens matter little once context geometry is correct.
  • Propose ReTime-Compose: roll out a frozen model once at logged speed, transport rollout frames (depth/poses) to candidate plan cameras to inject planned geometry, then complete only uncovered tokens; no training.

Limitation

The paper states that moving points are dropped on nuScenes and, on KITTI, enter all plans alike for their speed retiming reference.

Abstract (from arXiv)

Plan-conditioned driving world models are used to compare candidate plans, so their forecast should change with the plan as the real scene would, yet widely used control metrics check only whether the camera follows the command. Recorded drives already contain real outcomes of plans that were never driven: retiming a log gives pose-matched images and LiDAR for other speeds, and a stereo rig records a laterally shifted view. Against these references, released world models move their camera but barely move the scene: their scene response, the share of the real change in distance that the forecast reproduces, reaches at most about a third of real video's, and the trajectory score cannot tell an executed speed command from an ignored one. Interventions inside the generator trace the failure to the plan's channel. The forecast geometry follows the context frames; once they carry the planned geometry, plan tokens add little. ReTime-Compose therefore delivers the plan as geometry, without training. It transports the frames of one logged-speed rollout to the planned cameras, and the frozen model completes only what no frame covers. Frozen generators then follow a 0.54 m lateral shift nearly as the real scene does on KITTI, nuScenes and Argoverse 2, and respond more than re-sampling to slow-down plans in three generators and to speed-up plans in two.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Driving World Models Move the Camera, Not the Scene

Yuqiao Feng, Xiaoyu Ao

First announced by arXiv: 2026-11-13 · Original record ↗

arxiv
2611.15260
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.15260v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Plan-conditioned driving world models are used to compare candidate plans, so their forecast should change with the plan as the real scene would, yet widely used control metrics check only whether the camera follows the command. Recorded drives already contain real outcomes of plans that were never driven: retiming a log gives pose-matched images and LiDAR for other speeds, and a stereo rig records a laterally shifted view. Against these references, released world models move their camera but barely move the scene: their scene response, the share of the real change in distance that the forecast reproduces, reaches at most about a third of real video's, and the trajectory score cannot tell an executed speed command from an ignored one. Interventions inside the generator trace the failure to the plan's channel. The forecast geometry follows the context frames; once they carry the planned geometry, plan tokens add little. ReTime-Compose therefore delivers the plan as geometry, without training. It transports the frames of one logged-speed rollout to the planned cameras, and the frozen model completes only what no frame covers. Frozen generators then follow a 0.54 m lateral shift nearly as the real scene does on KITTI, nuScenes and Argoverse 2, and respond more than re-sampling to slow-down plans in three generators and to speed-up plans in two.