Driving · 2026-10-15
VITAL: From Visual Motion to Precise Action
Ao Liang, Youquan Liu, Mingxiao Li, Wei Chow, Lingdong Kong, Yue Liao, Dongyue Lu, Yu Yang, Wei Yin, Runnan Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
VITAL is an action tokenizer for autonomous driving that predicts a visually derived motion scaffold plus a normalized residual encoding the trajectory error left after that scaffold. Using flow matching, it jointly predicts both components from the current image, ego state, and navigation command without generating future images at inference. It also supports bounded residual refinement after imitation learning.
Why it matters
The authors argue that world-action models often train action experts on trajectory targets defined independently of visual dynamics, creating a visual–action representation gap. VITAL makes visual motion the principal prediction target and isolates remaining metric corrections in a residual representation, enabling one-step action generation and post-training refinement confined to residual coordinates.
Method
- Constructs an action tokenizer that pairs a visual motion scaffold (decoded from recorded future image tokens) with a compact metric residual encoding e* = vec(τ − τv*), with normalization so corrections have their own learning scale.
- Trains an action expert with flow matching to jointly predict normalized scaffold+residual codes from current image, ego state, and navigation command, using one flow step to decode an executable trajectory without future-image generation at inference.
- After imitation learning, freezes the generator/codec and learns bounded updates in residual coordinates (bounded by a radial limiter and a fixed correction basis) for residual reinforcement refinement with one deterministic correction at deployment.
Limitation
The authors state that the bounded correction construction “does not guarantee that every correction improves reward or satisfies task-level safety for every individual correction.”
Abstract (from arXiv)
World-action models aim to turn visual anticipation into driving decisions, yet typically train action experts on trajectory representations defined independently of visual dynamics. This motivates deriving action targets directly from visual motion. We first examine a simpler setting in which trajectories are reconstructed from recorded future images. Even without the need for future prediction, reconstruction captures the dominant maneuver but leaves centimeter-scale errors. We therefore introduce VITAL (Vision-Induced Tokens for Action Learning), an action tokenizer that couples a visually derived motion scaffold with a compact residual encoding the trajectory error left by that scaffold. This representation makes visual motion the principal prediction target and gives small metric corrections a separate, normalized learning scale. Using flow matching, the action expert jointly predicts both components from the current image, ego state, and navigation command, without generating future images at inference. The same residual coordinates support bounded reinforcement post-training that refines execution while keeping the predicted visual scaffold fixed. With only one action denoising step, VITAL achieves 90.8 PDMS on NAVSIM v1 navtest, within 0.13 points of its ten-step score at 3.29x the end-to-end inference speed. Without retraining, it achieves 91.1 EPDMS on NAVSIM v2 navtest and 38.6 on navhard. Project page: https://worldbench.github.io/vital
Related papers
- Native Action-Prior Learning from Videos for World Action Models
- MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies
- CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight
- ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
- MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
VITAL: From Visual Motion to Precise Action
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.20604
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World-action models aim to turn visual anticipation into driving decisions, yet typically train action experts on trajectory representations defined independently of visual dynamics. This motivates deriving action targets directly from visual motion. We first examine a simpler setting in which trajectories are reconstructed from recorded future images. Even without the need for future prediction, reconstruction captures the dominant maneuver but leaves centimeter-scale errors. We therefore introduce VITAL (Vision-Induced Tokens for Action Learning), an action tokenizer that couples a visually derived motion scaffold with a compact residual encoding the trajectory error left by that scaffold. This representation makes visual motion the principal prediction target and gives small metric corrections a separate, normalized learning scale. Using flow matching, the action expert jointly predicts both components from the current image, ego state, and navigation command, without generating future images at inference. The same residual coordinates support bounded reinforcement post-training that refines execution while keeping the predicted visual scaffold fixed. With only one action denoising step, VITAL achieves 90.8 PDMS on NAVSIM v1 navtest, within 0.13 points of its ten-step score at 3.29x the end-to-end inference speed. Without retraining, it achieves 91.1 EPDMS on NAVSIM v2 navtest and 38.6 on navhard. Project page: https://worldbench.github.io/vital