Latent Dynamics · 2026-11-03

OpenNWM: A Generalist Navigation World Model with Latent Action Pretraining

Dujun Nie, Ruiqi Song, Zixuan Wang, Hanqing Wang, Yifan Han, Baiyong Ding, Wenzhao Zheng, Hangbin Wu, Long Chen

arXiv:2611.01844PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

OpenNWM is a generalist navigation world model that learns universal latent actions from large-scale action-free navigation videos, then aligns them with physical motions via action-conditioned post-training. It introduces NavAnywhere (17.5M frames from 15 sources) and evaluates improved visual prediction and downstream navigation performance on in-domain and out-of-domain benchmarks.

Why it matters

The authors target a key limitation of navigation world models: generalization is constrained by limited action-labeled data. By pretraining with action-free videos using learned universal latent actions, OpenNWM aims to improve both perceptual dynamics learning and navigation performance for open-world navigation.

Method

  • Learns Universal Navigation Latent Actions (UNLAs) from action-free videos via visual prediction pretraining, then grounds them with physical-motion supervision when available.
  • Trains a latent-conditioned navigation world model using a diffusion-parameterized conditional distribution over future visual states, with latent actions serving as a shared interface for multimodal dynamics.
  • Performs receding-horizon planning by optimizing over real waypoint-action sequences using action-conditioned future visual rollouts and an energy combining goal similarity and navigation constraints.
Abstract (from arXiv)

Navigation world models predict future observations conditioned on navigation actions to support planning, but their generalization across real-world environments remains constrained by the limited coverage of action-labeled data. Action-free navigation videos provide diverse visual experience, yet lack the recorded actions required for action-conditioned training. We introduce OpenNWM, a generalist navigation world model with latent action pretraining. It learns universal latent actions from large-scale action-free videos through visual prediction pretraining, and further aligns them with physical motions via action-conditioned post-training. To support video pretraining, we curate NavAnywhere, a large-scale action-free navigation video dataset comprising 17.5 million visual observations from 15 heterogeneous sources. It captures navigation by ground robots, humans, and drones, with diverse viewpoints and motion patterns across five broad scene types: residential areas, public and commercial spaces, urban settings, parks, and natural landscapes. Experiments demonstrate improved visual prediction on in-domain and out-of-domain benchmarks, together with improved navigation performance. These results show that latent-action pretraining enables diverse, action-free videos to improve dynamics learning and generalization for open-world navigation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

OpenNWM: A Generalist Navigation World Model with Latent Action Pretraining

Dujun Nie, Ruiqi Song, Zixuan Wang, Hanqing Wang, Yifan Han, Baiyong Ding, Wenzhao Zheng, Hangbin Wu, Long Chen

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.01844
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.01844v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Navigation world models predict future observations conditioned on navigation actions to support planning, but their generalization across real-world environments remains constrained by the limited coverage of action-labeled data. Action-free navigation videos provide diverse visual experience, yet lack the recorded actions required for action-conditioned training. We introduce OpenNWM, a generalist navigation world model with latent action pretraining. It learns universal latent actions from large-scale action-free videos through visual prediction pretraining, and further aligns them with physical motions via action-conditioned post-training. To support video pretraining, we curate NavAnywhere, a large-scale action-free navigation video dataset comprising 17.5 million visual observations from 15 heterogeneous sources. It captures navigation by ground robots, humans, and drones, with diverse viewpoints and motion patterns across five broad scene types: residential areas, public and commercial spaces, urban settings, parks, and natural landscapes. Experiments demonstrate improved visual prediction on in-domain and out-of-domain benchmarks, together with improved navigation performance. These results show that latent-action pretraining enables diverse, action-free videos to improve dynamics learning and generalization for open-world navigation.