Driving · 2026-11-03
Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving
Xingtai Gui, Xubin Zhou, Yucheng Zhou, Dongqian Guo, Jiahao Gong, Feiyang Tan, Jianbing Shen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ScaleDrive is a scalable self-supervised framework for end-to-end autonomous driving that pretrains on large-scale internet open-world driving videos using action-free video latent prediction. The pretrained video encoder is kept frozen during downstream planning to test transferability. Scaling up action-free pretraining improves downstream closed-loop planning on NAVSIM/HUGSIM/Bench2Drive, and probes indicate increasing temporal-motion and spatial-geometric information in the frozen representation.
Why it matters
The authors study whether scaling an action-free video latent prediction objective improves transferable representations for downstream end-to-end planning. By freezing the encoder during policy training and evaluating across multiple planner architectures and closed-loop benchmarks, the work directly targets generalization and transfer rather than only representation quality in isolation.
Method
- Pretrain ScaleDrive on large-scale internet driving videos with action-free, latent prediction using a JEPA-style architecture with a context encoder, momentum target encoder, and L1 loss over masked latent tokens.
- Use combined patch-level spatial masking and complete temporal-slice masking to strengthen temporal modeling and reduce reliance on same-frame spatial shortcuts.
- For downstream planning, discard the predictor and keep the pretrained video encoder frozen while training only the planning module, enabling assessment of representation transfer.
Limitation
ScaleDrive focuses on learning transferable representations from monocular driving videos and does not explicitly model cross-view correspondences during pretraining. The authors also note that experiments use a fixed temporal sampling rate and clip length, leaving adaptive temporal sampling and longer-context pretraining unexplored.
Abstract (from arXiv)
End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.
Related papers
- MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
- Think Densely, Act Sparsely: Latent Expert Cognitive Chains for Vision-Language-Action Autonomous Driving
- GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation
- Closed-Loop Aware Planning via Imitation-Free Speed Scoring for End-to-End Autonomous Driving
- VITAL: From Visual Motion to Precise Action
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.02853
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.02853v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.