Driving · 2026-11-03
INTERACT: Learning Interaction-Aware World Dynamics from Driving Videos for Autonomous Driving
Zhan Chen, Songyan Zhang, Wenhui Huang, Zirui Li, Zhidong Wang, Qihang Huang, Shiyu Fang, Chen Lv
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
INTERACT learns an interaction-aware, action-conditioned world model from unlabeled front-view driving videos. It uses recurrent object-centric slots (Stage I), infers latent ego actions from visual transitions and slot context while predicting future slots (Stage II), then distills the learned dynamics into a trajectory planner with multimodal hypotheses (Stage III). Experiments on NAVSIM and nuScenes show state-of-the-art NAVSIM performance, with improvements on safety-critical collision metrics.
Why it matters
The authors target a planning need for anticipating how other traffic participants evolve under ego motion, without relying on object-level annotations. By combining recurrent slots for structured scene entities with latent ego-action inference and action-conditioned future slot prediction, INTERACT provides an interface between interaction-aware dynamics learning from raw videos and trajectory planning.
Method
- Stage I: bind frozen DINOv2 patch features into temporally persistent recurrent slots via recurrent Slot Attention, supervised with feature/temporal/appearance objectives.
- Stage II: infer latent ego action tokens from motion-aware V-JEPA2 transition cues and historical slots using MFormer + residual VQ; predict future slot evolution conditioned on inferred actions.
- Stage III: distill learned slots/actions/future-slot predictions into a DETR-style trajectory planner that outputs 64 trajectory hypotheses with action and future-slot guidance; train with WTA imitation plus world-model alignment.
Abstract (from arXiv)
Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.
Related papers
- ReWAM: Reciprocal World Action Models for Interactive Autonomous Driving
- DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
- PhysWAM: Physically Consistent World Action Model for Autonomous Driving
- MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
- BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-02 · Indexed by Pub-AI: 2026-11-03 UTC
INTERACT: Learning Interaction-Aware World Dynamics from Driving Videos for Autonomous Driving
Preprint posted: 2026-11-02 · Original record ↗
- arxiv
- 2611.02526
Explicit source versions (1)
- v1 · 2026-11-02
- Preprint available
Source date: 2026-11-02 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.02526v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.
