Latent Dynamics · 2026-10-20
Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning
Yanbo Wang, Nan Lin, Wendelin B\"ohmer, Javier Alonso-Mora
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Anchor-WM uses pretrained DINO features only as training-time supervision to learn a compact latent world model for action-conditioned prediction and visual planning. A predictive objective learns dynamics in the compact latent, while a training-only distillation head reconstructs dense DINO patch features to discourage representational collapse. At inference, DINO is discarded and planning runs entirely in the learned latent space, yielding strong success rates and faster planning than DINO-WM.
Why it matters
The authors address a design choice in visual world models: what information the latent state should preserve. They propose using pretrained visual features as teachers (training-time targets) rather than as the state space for prediction/planning. This aims to keep the visual prior and mitigate collapse, while avoiding the inference-time cost of propagating high-dimensional pretrained features during sampling-based planning.
Method
- Learn a compact latent state z_t with action-conditioned dynamics, trained with a latent prediction objective in the learned latent space.
- Use a frozen pretrained DINOv2 encoder only during training: a lightweight, training-only distillation head reconstructs flattened post-LayerNorm DINO patch features from z_t.
- Discard the DINO encoder and distillation head at inference; perform planning by rolling out predicted latent states under candidate action sequences and minimizing distance to the goal latent.
Limitation
The authors state that Anchor-WM requires an auxiliary distillation objective and that its planning speed advantage comes from doing planning in the compact learned latent space rather than directly in the high-dimensional DINO feature space.
Abstract (from arXiv)
A key design choice in visual world models is what information the latent state is encouraged to preserve. Models that learn a latent space with only a latent-prediction objective adapt to environment dynamics, but the predictive objective alone admits collapsed solutions and provides no explicit constraint on the visual information retained by the latent. Models built directly on pretrained visual features inherit a strong visual prior, but tie prediction and planning to a frozen state space. We propose Anchor-WM, which instead uses pretrained visual features as a teacher for learning a compact latent state rather than as the world-model state itself. Anchor-WM learns this compact latent with action-conditioned dynamics, while distilling the dense DINO representations through a training-only distillation head. This encourages the compact latent to retain visual information captured by the pretrained representation while discouraging representational collapse. At inference, the DINO encoder and distillation head are discarded, allowing prediction and planning to operate entirely in the learned latent space. Anchor-WM achieves the best average success rate across the four evaluated environments while enabling substantially faster planning than world models operating directly on DINO patch features. These results show that pretrained visual features can effectively guide world-model learning without defining the state space used for prediction and planning.
Related papers
- MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
- Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planning
- Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
- SkipWorld: Latent Skip Planning with Distilled World Models for Efficient Visual Navigation
- The Planning Limits of Latent World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-20 UTC
Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.23984
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
A key design choice in visual world models is what information the latent state is encouraged to preserve. Models that learn a latent space with only a latent-prediction objective adapt to environment dynamics, but the predictive objective alone admits collapsed solutions and provides no explicit constraint on the visual information retained by the latent. Models built directly on pretrained visual features inherit a strong visual prior, but tie prediction and planning to a frozen state space. We propose Anchor-WM, which instead uses pretrained visual features as a teacher for learning a compact latent state rather than as the world-model state itself. Anchor-WM learns this compact latent with action-conditioned dynamics, while distilling the dense DINO representations through a training-only distillation head. This encourages the compact latent to retain visual information captured by the pretrained representation while discouraging representational collapse. At inference, the DINO encoder and distillation head are discarded, allowing prediction and planning to operate entirely in the learned latent space. Anchor-WM achieves the best average success rate across the four evaluated environments while enabling substantially faster planning than world models operating directly on DINO patch features. These results show that pretrained visual features can effectively guide world-model learning without defining the state space used for prediction and planning.