Latent Dynamics · 2026-10-02

Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis

arXiv:2610.01942 PDF Code

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Latent-Foresight is an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model for forecasting in VFM feature space. It avoids two-stage pipelines by shaping the latent space for temporal predictability, using design choices to prevent latent collapse (stop-gradient targets/noisy inputs, latent normalization, auxiliary reconstruction on denoised latents, and noise scheduling). The authors report consistent gains over prior two-stage baselines across multiple tasks and horizons.

Why it matters

The paper addresses a mismatch in existing VFM-based world models: tokenizers trained only for reconstruction may not yield latent spaces that are predictable under generative temporal dynamics. By training tokenizer + flow predictor jointly (while preventing collapse), the authors aim to produce latents better suited for multi-task future scene understanding, potentially improving both forecasting accuracy and scalability by using a single training stage.

Method

Train an end-to-end VFM tokenizer (encoder + decoder) and a flow-matching latent dynamics predictor, while keeping the DINOv2 ViT backbone frozen.
Stabilize joint optimization with stop-gradient operations in the flow objective (on future latent targets and on noisy inputs), latent normalization (BatchNorm without affine), auxiliary reconstruction on predicted latents, and a selected noise schedule.
Evaluate by forecasting latent features and using frozen DPT heads for semantic segmentation, depth, and surface normals across multiple datasets and prediction horizons.

Limitation

The autoencoder architecture is kept lightweight; future work could explore more expressive tokenizer architectures and strategies for reducing the number of spatial tokens (e.g., learned spatial pooling or cross-attention-based compression) to enable more computationally efficient generative modeling at higher resolutions.

Abstract (from arXiv)

Predicting the future evolution of a scene is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models (VFMs) yields semantically rich representations that support diverse future scene understanding tasks. However, existing approaches rely on two-stage pipelines, where VFM features are first compressed using fixed dimensionality reduction (e.g., PCA) or independently trained autoencoders, and a separate predictor is trained on top of the resulting frozen latent space. This decoupling between representation learning and temporal prediction, as well as approaches that apply predictors directly on raw VFM features, provides no guarantee that the latent space is structured for predictable dynamics. In this work, we propose Latent-Foresight, an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model, explicitly shaping the representation to support temporal predictability. To enable stable joint optimization, we introduce several key design choices that prevent latent collapse and align reconstruction with generative objectives. Extensive experiments show that our approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation. We provide the implementation code and model weights at https://github.com/Sta8is/Latent-Foresight

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC

preprint