Driving · 2026-10-20
ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Yihui Yang, Yuanhong Yu, Jiawei Wang, Zhongrui Yu, Jia Shi, Kun Zhan, Ruizhen Hu, Xiaowei Zhou, Yu Liu, Sida Peng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ScaleNav targets language-conditioned local navigation to a visible text-specified target. It learns goal-directed visual dynamics from action-free Internet egocentric videos via instruction-conditioned video pretraining, aligns action prediction using action-labeled simulation, then jointly fine-tunes. The unified world-action model predicts future observations and executable action chunks, and is evaluated in simulation and on a real quadruped robot.
Why it matters
The authors address the data gap for end-to-end language-conditioned navigation by leveraging Internet egocentric videos, which provide scalable real-world visual experience but lack robot action labels. ScaleNav transfers navigation priors from video pretraining into executable action chunks using a three-stage training scheme, and reports improvements in navigation success on two simulation benchmarks and success on a real-robot deployment.
Method
- Unified world-action model with a shared DiT backbone that jointly predicts future observation tokens and executable action chunks conditioned on instruction, current observation, and history.
- Three-stage training: (1) instruction-conditioned video pretraining with no action tokens using a video flow-matching objective, (2) action alignment on action-labeled simulation with part of the model frozen, and (3) joint fine-tuning unfreezing remaining DiT modules and the video head.
- Instruction-conditioned video pretraining uses an initial-frame anchor plus recent history with chunk-wise causal attention; the robot iteratively predicts an action chunk from updated observations until it reaches the target and stops.
Abstract (from arXiv)
We study language-conditioned local navigation, where a robot must navigate to a text-specified target visible in its initial observation and stop at the described location. End-to-end navigation policies require diverse training data to generalize across scenes, yet real-robot data is costly to collect and simulated data leaves a sim-to-real gap. Internet egocentric videos offer a scalable source of real-world navigation experience. Our key insight is to learn goal-directed visual dynamics from Internet egocentric videos, and use these dynamics to support robot action generation. To this end, we propose ScaleNav, a unified world-action framework with three-stage training. We first perform instruction-conditioned video pretraining on these videos to learn how observations evolve as the camera approaches instructed targets across diverse real-world scenes. Action alignment then uses action-labeled simulated data to train the model to predict executable action chunks. Joint fine-tuning subsequently optimizes all trainable components together for navigation. ScaleNav achieves success rates of 86.5% on a single-view adaptation of Short-Horizon OVON and 94.9% on our HabitatGS-LN benchmark, outperforming the evaluated baselines on both benchmarks. We further deploy and evaluate ScaleNav on a real quadruped robot, with results supporting the model's ability to generalize to real-world navigation. Code and videos are available at https://zju3dv.github.io/ScaleNav/
Related papers
- Native Action-Prior Learning from Videos for World Action Models
- DR-VLA: Learning Human-Like Quadruped Navigation from Egocentric Web Videos via Domain-Invariant Scene Representations
- LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC
ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Preprint posted: 2026-10-19 · Original record ↗
- arxiv
- 2610.25690
Explicit source versions (1)
- v1 · 2026-10-19
- Preprint available
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We study language-conditioned local navigation, where a robot must navigate to a text-specified target visible in its initial observation and stop at the described location. End-to-end navigation policies require diverse training data to generalize across scenes, yet real-robot data is costly to collect and simulated data leaves a sim-to-real gap. Internet egocentric videos offer a scalable source of real-world navigation experience. Our key insight is to learn goal-directed visual dynamics from Internet egocentric videos, and use these dynamics to support robot action generation. To this end, we propose ScaleNav, a unified world-action framework with three-stage training. We first perform instruction-conditioned video pretraining on these videos to learn how observations evolve as the camera approaches instructed targets across diverse real-world scenes. Action alignment then uses action-labeled simulated data to train the model to predict executable action chunks. Joint fine-tuning subsequently optimizes all trainable components together for navigation. ScaleNav achieves success rates of 86.5% on a single-view adaptation of Short-Horizon OVON and 94.9% on our HabitatGS-LN benchmark, outperforming the evaluated baselines on both benchmarks. We further deploy and evaluate ScaleNav on a real quadruped robot, with results supporting the model's ability to generalize to real-world navigation. Code and videos are available at https://zju3dv.github.io/ScaleNav/