Video World Models · 2026-10-23
Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning
Yifeng Yao, Jinghan Li, Zhihao Zhang, Yajiao Xiong, Yang Jin, Kun Xu, Huishuai Zhang, Dongyan Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
VANE is a self-supervised visual pretraining framework that learns by next-frame prediction in a frozen semantic embedding space. A causal encoder summarizes past frames; a state predictor infers the representation of the entire next frame. An auxiliary generative decoder reconstructs future-frame patches to preserve fine-grained spatial detail. The same objective is extended to still images via local-to-global crop sequences, enabling unified image–video training.
Why it matters
The authors address a gap between masked/joint-embedding methods and autoregressive next-token prediction, by revisiting the granularity (frame-level vs token-level) and the prediction space (semantic embedding). VANE’s next-frame embedding prediction aims to make temporal evolution central while retaining spatial detail, and it is evaluated as a unified image–video pretraining objective across anticipation, recognition, and dense perception benchmarks.
Method
- Next-frame prediction in embedding space: predict the frozen semantic tokenizer embedding of the entire next frame from past frames.
- Causal encoder with frame-causal attention (unidirectional across frames) to avoid future leakage; prediction tokens attend only to history under target isolation.
- Auxiliary generative decoder reconstructs future-frame patches (flow matching) conditioned on the predicted representation; unify images and videos using local-to-global crop sequences.
Abstract (from arXiv)
Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. This prediction is carried out in a fixed embedding space provided by a frozen semantic tokenizer, emphasizing scene content and structure. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.
Related papers
- Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
- MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
- Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation
- Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning
- ALPS: Visual State Encoders for World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC
Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning
Preprint posted: 2026-10-22 · Original record ↗
- arxiv
- 2610.30724
Explicit source versions (1)
- v1 · 2026-10-22
- Preprint available
Source date: 2026-10-22 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. This prediction is carried out in a fixed embedding space provided by a frozen semantic tokenizer, emphasizing scene content and structure. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.