Latent Dynamics · 2026-10-01
Social-WM: Safety-Aware Latent World Models for Robot Social Navigation
Zhihao Zheng, Mooi Choo Chuah
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Social-WM is an action-conditioned latent world-model planning framework for robot social navigation from egocentric RGB video. It distinguishes nominal actions from realizable actions using safety-aware future prediction (targeting the actual observed future) plus a realizable inverse-dynamics objective. During planning, it imagines action chunks and uses nominal–realizable discrepancy as a safety signal to select safer candidates.
Why it matters
The authors address a key gap in social navigation: a nominal command may look feasible but becomes constrained or blocked by pedestrians and obstacles. Social-WM learns these safety-relevant consequences from offline RGB trajectories and uses a realizability estimate before execution, aiming to reduce human collisions and improve personal-space compliance without explicit pedestrian tracking, privileged human state, or online RL.
Method
- Trains a frozen DINOv2-small visual encoder and a transformer latent predictor for action-conditioned future prediction, where the target is the actual observed future after each nominal action command.
- Introduces a realizable inverse-dynamics objective that supervises inverse dynamics to recover the action actually executed (realizable action) from latent transitions rather than the issued nominal command.
- At deployment, proposes K action chunks with a goal-independent CVAE, imagines rollouts in latent space, estimates realizability for imagined transitions, and uses nominal–realizable discrepancy to filter admissible candidates for closed-loop selection.
Limitation
The authors state that many remaining failures are timeouts: the current world model mainly reasons about short-horizon safety and action realizability, while long-horizon route selection remains unchanged. They also note long-horizon challenges for image-goal navigation when goal content is not yet visible or after large deviations.
Abstract (from arXiv)
Safe social navigation requires a robot to anticipate not only the future consequences of its actions, but also whether a nominal action can actually be executed under surrounding physical and social constraints. We present Social-WM, an efficient latent world-model planning framework trained from egocentric RGB video sequences. Our key observation is that social-navigation experience contains a systematic discrepancy between the nominal action and the realizable action: a nominal forward action may be fully executed in free space, but needs to be constrained when heading towards a pedestrian or obstacle. Social-WM learns these safety-relevant consequences directly through action-conditioned future prediction, where the target is the actual observed future following each command. We further introduce a realizable inverse-dynamics objective that associates observed latent transitions with the action actually realized rather than the nominal one. At deployment, candidate actions are imagined through the latent world model, and the inverse dynamics model estimates their realizability; nominal--realizable discrepancy then provides a safety signal before execution. The learned dynamics and realizability model remain goal-independent and support both position- and image-goal navigation. On Social-HM3D, Social-WM achieves 63.77% success while reducing human collisions to 21.67%, and maintains strong performance under zero-shot transfer to Social-MP3D, without explicit pedestrian tracking, privileged human state, or online reinforcement learning.
Related papers
- Seeing Beyond the Next Step: World-Model-Guided Human-Like Navigation in Multi-Agent Scenes
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
- ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC
Social-WM: Safety-Aware Latent World Models for Robot Social Navigation
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2609.40177
Explicit source versions (1)
- v2 · 2026-10-01
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v2
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
Safe social navigation requires a robot to anticipate not only the future consequences of its actions, but also whether a nominal action can actually be executed under surrounding physical and social constraints. We present Social-WM, an efficient latent world-model planning framework trained from egocentric RGB video sequences. Our key observation is that social-navigation experience contains a systematic discrepancy between the nominal action and the realizable action: a nominal forward action may be fully executed in free space, but needs to be constrained when heading towards a pedestrian or obstacle. Social-WM learns these safety-relevant consequences directly through action-conditioned future prediction, where the target is the actual observed future following each command. We further introduce a realizable inverse-dynamics objective that associates observed latent transitions with the action actually realized rather than the nominal one. At deployment, candidate actions are imagined through the latent world model, and the inverse dynamics model estimates their realizability; nominal--realizable discrepancy then provides a safety signal before execution. The learned dynamics and realizability model remain goal-independent and support both position- and image-goal navigation. On Social-HM3D, Social-WM achieves 63.77% success while reducing human collisions to 21.67%, and maintains strong performance under zero-shot transfer to Social-MP3D, without explicit pedestrian tracking, privileged human state, or online reinforcement learning.
