Driving · 2026-11-03

INTERACT: Learning Interaction-Aware World Dynamics from Driving Videos for Autonomous Driving

Zhan Chen, Songyan Zhang, Wenhui Huang, Zirui Li, Zhidong Wang, Qihang Huang, Shiyu Fang, Chen Lv

arXiv:2611.02526PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

INTERACT learns an interaction-aware, action-conditioned world model from unlabeled front-view driving videos. It uses recurrent object-centric slots (Stage I), infers latent ego actions from visual transitions and slot context while predicting future slots (Stage II), then distills the learned dynamics into a trajectory planner with multimodal hypotheses (Stage III). Experiments on NAVSIM and nuScenes show state-of-the-art NAVSIM performance, with improvements on safety-critical collision metrics.

Why it matters

The authors target a planning need for anticipating how other traffic participants evolve under ego motion, without relying on object-level annotations. By combining recurrent slots for structured scene entities with latent ego-action inference and action-conditioned future slot prediction, INTERACT provides an interface between interaction-aware dynamics learning from raw videos and trajectory planning.

Method

  • Stage I: bind frozen DINOv2 patch features into temporally persistent recurrent slots via recurrent Slot Attention, supervised with feature/temporal/appearance objectives.
  • Stage II: infer latent ego action tokens from motion-aware V-JEPA2 transition cues and historical slots using MFormer + residual VQ; predict future slot evolution conditioned on inferred actions.
  • Stage III: distill learned slots/actions/future-slot predictions into a DETR-style trajectory planner that outputs 64 trajectory hypotheses with action and future-slot guidance; train with WTA imitation plus world-model alignment.
Abstract (from arXiv)

Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-02 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

INTERACT: Learning Interaction-Aware World Dynamics from Driving Videos for Autonomous Driving

Zhan Chen, Songyan Zhang, Wenhui Huang, Zirui Li, Zhidong Wang, Qihang Huang, Shiyu Fang, Chen Lv

Preprint posted: 2026-11-02 · Original record ↗

arxiv
2611.02526
Explicit source versions (1)
  • v1 · 2026-11-02
  1. Preprint available

    Source date: 2026-11-02 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.