Driving · 2026-10-09
VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving
Zhaoyang Liu, Kun Jiang, Ziying Song, Diange Yang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
VGGTWorld-VLA extends VGGT-World with driving-intention control for autonomous driving. It injects ego action and VLA-derived semantic tokens into the future-token stream via a geometry–language–action bridge and layer-wise conditioning, enabling different future 3D geometry predictions from the same observation under different ego actions. Evaluated on NAVSIM for future geometry/depth prediction, the authors report improved Token MSE, RMSE, and δ1 versus VGGT-World.
Why it matters
The authors address a limitation of VGGT-World where future evolution is primarily driven by historical observations and is not explicitly conditioned on driving intentions/actions, making it hard to represent alternative action-dependent physical futures. By adding semantic and action conditioning while keeping VGGT-World as the backbone, VGGTWorld-VLA aims to produce controllable future 3D evolution consistent with intended maneuvers.
Method
- Extend VGGT-World with an intention-conditioned geometry world model: action–semantic conditioning injected into the future-token stream via separate residual cross-attention modules.
- Introduce a geometry–language–action bridge that combines historical geometry, multi-representation ego actions (maneuver + trajectory/velocity), and VLA semantic tokens from multi-camera observations and a textual prompt.
- Use conditional flow matching to predict future VGGT geometry tokens under conditions, then decode to future depth/3D; supervise with VGGT-derived pseudo targets for depth (and point outputs).
Abstract (from arXiv)
VGGT provides a strong foundation for geometry-centric world models by recovering unified 3D scene geometry from visual observations. Although recent extensions enable temporal 3D prediction, their future evolution remains weakly conditioned on driving intentions and actions, limiting their ability to model alternative action-dependent futures. We propose VGGTWorld-VLA, an intention-conditioned extension of VGGT-World for controllable 3D world evolution in autonomous driving. First, we introduce an action--semantic conditioning mechanism that injects complementary driving semantics and ego-motion representations into the future-token stream, enabling different future geometry predictions for the same observed scene under alternative ego actions. Second, we develop a geometry--language--action bridge that adapts historical geometry, VLA semantic features, and maneuver and trajectory representations for joint conditioning of future geometry prediction. We evaluate future geometry prediction on NAVSIM, while conditioning ablations further examine the contributions of semantic and action information. Compared with the baseline, our method demonstrates competitive geometry prediction performance. Ablation studies further support the effectiveness of semantic and action conditioning. These results demonstrate the potential of semantic and action conditioning for controllable VGGT-based world prediction in autonomous driving.
Related papers
- Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving
- DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving
- AffordDrive3D: Affordance-Aware World-Action Modeling with Spatial Understanding
- DriveWorld3D Evidence Constrained 3D World Modeling and Generative Simulation for Real-World Autonomous Driving
- OccWAM: Occupancy World-Action Models with Joint Scene-to-Ego Flow Matching
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC
VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving
Preprint posted: 2026-10-08 · Original record ↗
- arxiv
- 2610.11161
Explicit source versions (1)
- v1 · 2026-10-08
- Preprint available
Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
VGGT provides a strong foundation for geometry-centric world models by recovering unified 3D scene geometry from visual observations. Although recent extensions enable temporal 3D prediction, their future evolution remains weakly conditioned on driving intentions and actions, limiting their ability to model alternative action-dependent futures. We propose VGGTWorld-VLA, an intention-conditioned extension of VGGT-World for controllable 3D world evolution in autonomous driving. First, we introduce an action--semantic conditioning mechanism that injects complementary driving semantics and ego-motion representations into the future-token stream, enabling different future geometry predictions for the same observed scene under alternative ego actions. Second, we develop a geometry--language--action bridge that adapts historical geometry, VLA semantic features, and maneuver and trajectory representations for joint conditioning of future geometry prediction. We evaluate future geometry prediction on NAVSIM, while conditioning ablations further examine the contributions of semantic and action information. Compared with the baseline, our method demonstrates competitive geometry prediction performance. Ablation studies further support the effectiveness of semantic and action conditioning. These results demonstrate the potential of semantic and action conditioning for controllable VGGT-based world prediction in autonomous driving.