Driving · 2026-11-06

Seeing Beyond the Next Step: World-Model-Guided Human-Like Navigation in Multi-Agent Scenes

Seong-Eun Hong, JuYeong Hwang, Jinhyun Kim, Hanyoung Jang, HyeongYeop Kang

arXiv:2611.07730PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors train a fast navigation policy using two action-conditioned world models only at training time. They decompose consequence prediction into social dynamics (relative position/velocity with direction) and scene semantics (navigation-relevant segmentation with pedestrians ignored). GRPO distills group-relative preference scores over candidate motion anchors into the policy, so at deployment the policy acts in a single forward pass without world-model rollouts.

Why it matters

The paper targets a deployability trade-off: world-model foresight is expensive if rolled out at every decision step. By amortizing world-model predictions into a preference-distilled anchor policy, the authors aim to improve navigation safety and social/goal coherence while keeping inference-time cost independent of world-model rollout details.

Method

  • Train an action prior over K=100 motion anchors built offline with K-means on 20-frame, 10-frame-stride demonstration clips (25 FPS), then supervise the anchor classifier with the nearest demonstrated anchor.
  • Use two frozen, anchor-conditioned predictors during refinement: a State Transition World Model for next relative position–velocity (with an explicit direction term) and a Semantic Transition World Model predicting navigation-relevant segmentation classes while assigning pedestrians an ignore label in the target.
  • For each observation, score 10 candidate anchors (top-5 policy anchors + 5 random) using goal progress, smoothness, immediate and future interaction risk, and semantic validity; refine the policy with GRPO (prefer candidates whose score exceeds the group mean). Remove world models at deployment for single forward-pass action selection.

Limitation

The reconstructed Unity scenes are built on commercially licensed assets that cannot be redistributed, so they are not released.

Abstract (from arXiv)

Selecting a next motion in a populated scene is not just a matter of avoiding what is currently in the way: an action can be locally feasible yet drift onto the road instead of the crosswalk, cut into a stream of crossing pedestrians, or bottleneck a narrow passage. These failure modes are invisible from the current state alone, and acting on them requires reasoning about what each candidate action will cause. World models offer exactly this, namely action-conditioned predictions of the future, but rolling them out at every decision step is a cost deployable navigation systems cannot pay, and the field has largely traded foresight for speed. We argue this is a false dichotomy. The natural place for a world model in human-aware navigation is at training time, as a preference oracle that distills foresight into a fast policy, rather than at inference time, as an online planner. We pair a discrete visual action prior with two task-aligned world models, one for nearby social dynamics and one for future scene semantics, that score candidate actions by their predicted consequences and refine the policy through group-relative preference distillation. At deployment, both world models are removed and the policy acts in a single forward pass. Across five held-out evaluation scenarios, this yields navigation that is simultaneously safer, more socially and group-coherent, and more goal-directed than strong reactive and learned baselines, with naive observers rating the resulting motion as more human-like. Taken together, our findings point to a different approach for bringing world-model foresight into deployable agents: not accelerating rollouts, but amortizing them into the policy so that the cost of looking ahead is confined to training. Project page: https://seoneun.github.io/-NeurIPS-2026-Seeing-Beyond-the-Next-Step/

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: Date not supplied · Indexed by Pub-AI: 2026-11-06 UTC

preprint

Seeing Beyond the Next Step: World-Model-Guided Human-Like Navigation in Multi-Agent Scenes

Seong-Eun Hong, JuYeong Hwang, Jinhyun Kim, Hanyoung Jang, HyeongYeop Kang

Source date: Date not supplied · Original record ↗

arxiv
2611.07730

    Imported from the existing registry. No new publication milestone asserted.

    Abstract · arxiv

    Selecting a next motion in a populated scene is not just a matter of avoiding what is currently in the way: an action can be locally feasible yet drift onto the road instead of the crosswalk, cut into a stream of crossing pedestrians, or bottleneck a narrow passage. These failure modes are invisible from the current state alone, and acting on them requires reasoning about what each candidate action will cause. World models offer exactly this, namely action-conditioned predictions of the future, but rolling them out at every decision step is a cost deployable navigation systems cannot pay, and the field has largely traded foresight for speed. We argue this is a false dichotomy. The natural place for a world model in human-aware navigation is at training time, as a preference oracle that distills foresight into a fast policy, rather than at inference time, as an online planner. We pair a discrete visual action prior with two task-aligned world models, one for nearby social dynamics and one for future scene semantics, that score candidate actions by their predicted consequences and refine the policy through group-relative preference distillation. At deployment, both world models are removed and the policy acts in a single forward pass. Across five held-out evaluation scenarios, this yields navigation that is simultaneously safer, more socially and group-coherent, and more goal-directed than strong reactive and learned baselines, with naive observers rating the resulting motion as more human-like. Taken together, our findings point to a different approach for bringing world-model foresight into deployable agents: not accelerating rollouts, but amortizing them into the policy so that the cost of looking ahead is confined to training. Project page: https://seoneun.github.io/-NeurIPS-2026-Seeing-Beyond-the-Next-Step/