Video World Models · 2026-11-13

Prediction or Persistence? What Future-Prediction Scores Miss About World Models

Animesh Maheshwari

arXiv:2611.15082PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors argue that “future-prediction” scores for pretrained video world models mix encoder quality, near-future similarity to the recent past, and the predictor’s own contribution. They add a matched persistence control (predictor replaced by identity) and show predictor contribution can be positive, small, or negative even when the original score is high. They recommend reporting future-prediction scores with their persistence baseline.

Why it matters

If model selection or evaluation relies mainly on future-prediction scores, those numbers may not reflect prediction. By isolating the predictor via a predictor-disabled persistence control, the study makes future-prediction scores interpretable: the gap between the score and persistence baseline is what indicates the predictor’s contribution. The authors also note that downstream control/planning usefulness is a separate question.

Method

  • Future-retrieval probe on 27,000 zero-shot trials: pick the true continuation among 11 candidates from the same household egocentric video, ranked by the model’s own (native-energy) criterion.
  • Matched persistence control: pair each model with an identical copy where the predictor is replaced by identity (predictor never invoked), so the top-1 difference isolates predictor contribution (for one extra forward pass).
  • Use video-level cluster bootstrap confidence intervals and sweep input span A and rollout horizon W, evaluating within an “envelope” constrained by positional addressing capacity for frozen predictors.

Limitation

The probe is on passive household video (no robot in the loop), so it does not measure control, planning, or downstream robot performance. Retrieval among 11 candidates is easier than generation. Also, DINO-WM and LeWM are action-conditioned models trained on simulated PushT; they are evaluated with zero actions through their own action encoders, outside their action regime.

Abstract (from arXiv)

Pretrained video world models are compared and chosen as components for robot learning largely by future-prediction scores. On its own, such a score cannot be read as evidence of prediction, because it mixes encoder quality, the near future's resemblance to the recent past, and the predictor's own contribution. We separate the three with a matched control. On 27,000 future-retrieval trials from household egocentric video, a model must pick the true continuation of a clip among eleven candidates from the same video (chance 0.091), ranked zero-shot by its own training criterion. Each model is paired with a persistence control, an identical copy whose predictor is never invoked, so the paired difference isolates the predictor. Score and contribution dissociate within the native-video V-JEPA family alone: the predictor adds +0.291 top-1 for V-JEPA 2.1 but +0.03 to +0.04 for V-JEPA 2 on the identical trials. Across five models the contribution ranges from +0.291 to negative, and it is negative only for the two action-conditioned models, which we evaluate outside their training regime. Persistence alone recovers the encoder ordering expected from pretraining, a sign that the probe tracks real signal. We suggest reporting each future-prediction score together with its persistence baseline. The gap between them, not the score, is what makes the number interpretable: a necessary check costing one forward pass. Establishing downstream usefulness is a separate matter, for a planning or control evaluation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Prediction or Persistence? What Future-Prediction Scores Miss About World Models

Animesh Maheshwari

First announced by arXiv: 2026-11-13 · Original record ↗

arxiv
2611.15082
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.15082v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Pretrained video world models are compared and chosen as components for robot learning largely by future-prediction scores. On its own, such a score cannot be read as evidence of prediction, because it mixes encoder quality, the near future's resemblance to the recent past, and the predictor's own contribution. We separate the three with a matched control. On 27,000 future-retrieval trials from household egocentric video, a model must pick the true continuation of a clip among eleven candidates from the same video (chance 0.091), ranked zero-shot by its own training criterion. Each model is paired with a persistence control, an identical copy whose predictor is never invoked, so the paired difference isolates the predictor. Score and contribution dissociate within the native-video V-JEPA family alone: the predictor adds +0.291 top-1 for V-JEPA 2.1 but +0.03 to +0.04 for V-JEPA 2 on the identical trials. Across five models the contribution ranges from +0.291 to negative, and it is negative only for the two action-conditioned models, which we evaluate outside their training regime. Persistence alone recovers the encoder ordering expected from pretraining, a sign that the probe tracks real signal. We suggest reporting each future-prediction score together with its persistence baseline. The gap between them, not the score, is what makes the number interpretable: a necessary check costing one forward pass. Establishing downstream usefulness is a separate matter, for a planning or control evaluation.