Video World Models · 2026-10-07

Tracking Is Not Permanence: What Video World Models Keep of a Hidden Object

Peng Xie, Amr Alanwar

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors probe permanence in frozen video world models by hiding an object from a V-JEPA 2 predictor and comparing its hidden-region prediction to the frozen encoder’s representations of two futures that differ only in the hidden region. V-JEPA 2 keeps a stationary hidden object partly, keeps a carried one not at all, and loses a moving one within 0.3s (0.5s with its tube mask). Training can install permanence via a prior from synthetic containers.

Why it matters

Video predictors are used as world models for planning, which requires remembering state beyond visibility. The paper shows that, in V-JEPA 2’s released predictor latents, permanence for hidden objects is limited: the decision drops quickly for moving and container-carried objects, yet the information exists in the encoder (decodable presence and contents). The authors also show training interventions can install a containment prior and raise IntPhys-2019, while the choice of scoring rule affects what the benchmark credits.

Method

Freeze V-JEPA 2 encoder and predictor; hide an object region after an event; let the frozen predictor infer the missing tokens and use a belief probe (two-world comparison) to read permanence per tubelet time slot.
Compare decoder/prediction readouts to matched controls (never-rendered, never-visible, exit/reveal) and to other probes: encoder linear/logistic probes, VideoMAE pixel reconstruction, and teacher-forced Cosmos next-token scoring.
Train interventions from released weights (temporally-partial masks, synthetic container curriculum, continued tube-mask pretraining, and real-hand three-world coin events) and evaluate with IntPhys-2019 (pairwise surprise-based accuracy).

Limitation

The authors state that they did not probe V-JEPA 2-AC, V-JEPA 2.1, diffusion world models, VideoMAE v2, or models with explicit memory (e.g., WorldMem, state-space world models, slot models). They also say their synthetic-scene permanence finding is not tested on real occlusion video with a two-world test (e.g., TCOW or the Perception Test).

Abstract (from arXiv)

Video world models track objects they can see; we ask what they keep of objects they cannot. We hide an object from a frozen V-JEPA 2 predictor and compare its prediction for the hidden region with the encoder's representation of two worlds that differ only inside that region. The predictor's decision keeps a stationary object in part and one carried inside a container not at all, and loses a moving one within 0.3 s (0.5 s under V-JEPA's own tube mask; ViT-H keeps it to 1.1 s at pretraining's 90% masking ratio); in projection a trace remains, below the midpoint, at 14-60% of what a baseline copying the last view retains. The information is there: the encoder reads the object's presence at 1.00 and keeps a closed container's contents decodable for 3.5 s, while the predictor's output, read with the encoder's own probe, contains the ball in 2% of scenes once the box has been closed for half a second. On rendered scenes, permanence is missing on the predictor's side, and training installs it cheaply as a prior: three thousand predictor-only steps on synthetic containers take this belief from 0.05 to 1.00 against two matched controls. They also raise IntPhys-2019 from 84.2% to 93.3%, but so does a curriculum without containers, and which training habit the benchmark credits changes with its scoring rule. Continued training with tube masks produces 1.1-1.6 s of moving-object carry-over on manipulation and internet-style video, so the deficit is not intrinsic to latent prediction. VideoMAE keeps almost nothing, and Cosmos's next-token prediction keeps a stationary hidden object but not one carried inside a moving container.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-07 UTC

preprint