Driving · 2026-10-26

See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space

Jiawei Chen, Yinxi Wang, Zihan Yang, Meijun Gao, Yaowei Liu, Shuming Zhang, Zecheng Li, Xiangju Mi, Yuxuan Liu, Runzhong Huang, Tianlin Li, Xun Zhang, Zhengyang Wang, Xu Miao, Tong Yang

arXiv:2610.31762PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

RepresentNav is a navigation foundation model that constructs a self-constructed representation space from frozen vision foundation models using Representation Residuals, and learns a latent chain-of-thought “dream” in that same space via JEPA training (next-representation prediction plus VISReg). The authors report state-of-the-art results on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, with gains transferring to a physical robot.

Why it matters

The paper argues that navigation policies should act on task-shaped representations rather than using a fixed encoder interface chosen before the task. RepresentNav builds a representation space dynamically from multiple foundation models and trains a latent world-model-like predictor in that same space, aiming for foresight for planning and improved control (e.g., reduced collisions).

Method

  • Constructs a self-constructed representation space using Representation Residuals: a compact three-layer transformer attending over aligned features spanning the full depth of three frozen vision foundation models (DINOv3, Depth Anything 3, OpenD4RT).
  • Uses a latent chain-of-thought rollout before every action: predicts future Representation Residual [CLS] target representations (0.4–1.6s ahead) and trains the dream end-to-end as a JEPA with a next-representation prediction loss and VISReg regularization.
  • Trains with a two-stage curriculum: first the policy acts through its dream (restricted causal attention), then jointly acts on what it sees and dreams together; navigation/path heads are trained alongside the dream targets.
Abstract (from arXiv)

What visual representation should a navigation policy act on? Navigation policies built on vision foundation models largely leave representation design to the encoders' pretraining objectives. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy's own objectives from vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On VLNVerse fine-grained, RepresentNav achieves 83.80% success, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot's own motion over the next second: the dream emerges as a compact latent world model.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-26 UTC

preprint

See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space

Jiawei Chen, Yinxi Wang, Zihan Yang, Meijun Gao, Yaowei Liu, Shuming Zhang, Zecheng Li, Xiangju Mi, Yuxuan Liu, Runzhong Huang, Tianlin Li, Xun Zhang, Zhengyang Wang, Xu Miao, Tong Yang

Preprint posted: 2026-10-22 · Original record ↗

arxiv
2610.31762
Explicit source versions (1)
  • v1 · 2026-10-22
  1. Preprint available

    Source date: 2026-10-22 · First detected: 2026-11-02 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

What visual representation should a navigation policy act on? Navigation policies built on vision foundation models largely leave representation design to the encoders' pretraining objectives. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy's own objectives from vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On VLNVerse fine-grained, RepresentNav achieves 83.80% success, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot's own motion over the next second: the dream emerges as a compact latent world model.