Latent Dynamics · 2026-10-01

Social-WM: Safety-Aware Latent World Models for Robot Social Navigation

Zhihao Zheng, Mooi Choo Chuah

arXiv:2609.40177PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Social-WM is an action-conditioned latent world-model planning framework for robot social navigation from egocentric RGB video. It distinguishes nominal actions from realizable actions using safety-aware future prediction (targeting the actual observed future) plus a realizable inverse-dynamics objective. During planning, it imagines action chunks and uses nominal–realizable discrepancy as a safety signal to select safer candidates.

Why it matters

The authors address a key gap in social navigation: a nominal command may look feasible but becomes constrained or blocked by pedestrians and obstacles. Social-WM learns these safety-relevant consequences from offline RGB trajectories and uses a realizability estimate before execution, aiming to reduce human collisions and improve personal-space compliance without explicit pedestrian tracking, privileged human state, or online RL.

Method

  • Trains a frozen DINOv2-small visual encoder and a transformer latent predictor for action-conditioned future prediction, where the target is the actual observed future after each nominal action command.
  • Introduces a realizable inverse-dynamics objective that supervises inverse dynamics to recover the action actually executed (realizable action) from latent transitions rather than the issued nominal command.
  • At deployment, proposes K action chunks with a goal-independent CVAE, imagines rollouts in latent space, estimates realizability for imagined transitions, and uses nominal–realizable discrepancy to filter admissible candidates for closed-loop selection.

Limitation

The authors state that many remaining failures are timeouts: the current world model mainly reasons about short-horizon safety and action realizability, while long-horizon route selection remains unchanged. They also note long-horizon challenges for image-goal navigation when goal content is not yet visible or after large deviations.

Abstract (from arXiv)

Safe social navigation requires a robot to anticipate not only the future consequences of its actions, but also whether a nominal action can actually be executed under surrounding physical and social constraints. We present Social-WM, an efficient latent world-model planning framework trained from egocentric RGB video sequences. Our key observation is that social-navigation experience contains a systematic discrepancy between the nominal action and the realizable action: a nominal forward action may be fully executed in free space, but needs to be constrained when heading towards a pedestrian or obstacle. Social-WM learns these safety-relevant consequences directly through action-conditioned future prediction, where the target is the actual observed future following each command. We further introduce a realizable inverse-dynamics objective that associates observed latent transitions with the action actually realized rather than the nominal one. At deployment, candidate actions are imagined through the latent world model, and the inverse dynamics model estimates their realizability; nominal--realizable discrepancy then provides a safety signal before execution. The learned dynamics and realizability model remain goal-independent and support both position- and image-goal navigation. On Social-HM3D, Social-WM achieves 63.77% success while reducing human collisions to 21.67%, and maintains strong performance under zero-shot transfer to Social-MP3D, without explicit pedestrian tracking, privileged human state, or online reinforcement learning.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC

preprint

Social-WM: Safety-Aware Latent World Models for Robot Social Navigation

Zhihao Zheng, Mooi Choo Chuah

Preprint posted: 2026-09-30 · Original record ↗

arxiv
2609.40177
Explicit source versions (1)
  • v2 · 2026-10-01
  1. Preprint available

    Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
  2. Explicit source version detected · v2

    Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Safe social navigation requires a robot to anticipate not only the future consequences of its actions, but also whether a nominal action can actually be executed under surrounding physical and social constraints. We present Social-WM, an efficient latent world-model planning framework trained from egocentric RGB video sequences. Our key observation is that social-navigation experience contains a systematic discrepancy between the nominal action and the realizable action: a nominal forward action may be fully executed in free space, but needs to be constrained when heading towards a pedestrian or obstacle. Social-WM learns these safety-relevant consequences directly through action-conditioned future prediction, where the target is the actual observed future following each command. We further introduce a realizable inverse-dynamics objective that associates observed latent transitions with the action actually realized rather than the nominal one. At deployment, candidate actions are imagined through the latent world model, and the inverse dynamics model estimates their realizability; nominal--realizable discrepancy then provides a safety signal before execution. The learned dynamics and realizability model remain goal-independent and support both position- and image-goal navigation. On Social-HM3D, Social-WM achieves 63.77% success while reducing human collisions to 21.67%, and maintains strong performance under zero-shot transfer to Social-MP3D, without explicit pedestrian tracking, privileged human state, or online reinforcement learning.