Latent Dynamics · 2026-11-03

When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?

Anton Hristov, Aleksandar Slavov, Vasil Vasilev

arXiv:2611.01815PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors argue that video world-model prediction error is not a reliable measure of physical surprise: it can rise or fall due to appearance, static content, motion, or ordinary prediction difficulty. Using IntPhys2, Physion++, and Kubric, they show appearance is easy to decode from frozen features, action sensitivity exists, but native prediction error can order physical-possibility outcomes inconsistently and can be lower for physically impossible events.

Why it matters

Many approaches interpret prediction error as “surprise” to detect physically unexpected events. This paper evaluates what latent features encode and how native prediction error changes under controlled interventions, showing that recoverable physical information and action sensitivity are insufficient for trustworthy physical-surprise claims.

Method

  • Evaluate frozen latent representations with linear probes for appearance and possible-vs-impossible events (IntPhys2), comparing V-JEPA2-G, VideoMAE2-G, and simple visual statistics (PixelSummary).
  • Analyze latent geometry across network depth to measure relative appearance vs physical changes and their alignment/overlap (IntPhys2).
  • Test native prediction error under controlled changes: threshold transfer (Kubric crossed-support), appearance/static controls (Physion++), and action vs appearance sensitivity (action-conditioned models on TwoRoom/PushT; Physion++).

Limitation

The authors note that their comparisons retain each evaluated pipeline’s encoders, token selection, normalization, and native distance, so architecture/training differences are not isolated to a specific objective or component, and the geometry-to-error regression provides limited evidence of added predictive value from tested linear geometry features.

Abstract (from arXiv)

Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?

Anton Hristov, Aleksandar Slavov, Vasil Vasilev

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.01815
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.01815v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.