Driving · 2026-10-26
Interpreting Physics in Video Encoders using Multi-Object Interaction Videos
Ayushi Mehrotra, Sai Vidyaranya Nuthalapati
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors probe whether two frozen self-supervised video encoders (V-JEPA2 and VideoMAE) encode relational physics in two-object interactions (momentum conservation, elasticity, kinetic-energy changes) and in cube–floor friction. They find elastic vs inelastic collisions are reliably separated in both, but momentum-violation and kinetic-energy transfer across environments are mixed, and friction-coefficient readouts do not survive geometry changes.
Why it matters
If a physical AI policy plans by rolling out a video-encoder latent, then what relational quantities (e.g., momentum transfer, restitution, friction) the latent represents determines what the agent can plan. The paper directly measures relational-physics attributes inside frozen encoders and tests whether such readouts transfer across scene changes, helping distinguish genuine physics information from visual correlates.
Method
- Use two frozen self-supervised video encoders (VideoMAE-base and V-JEPA2-ViT-L) and probe pooled activations with ridge-regression probes (continuous: out-of-fold R2; binary: out-of-fold AUC).
- Probe relational quantities for two-body collisions (momentum conservation, elasticity, kinetic-energy changes) and friction for cube–floor settings; test transfer by fitting probes in one scenario and applying unchanged in another.
- Use matched observers with access to visible kinematic scalars as controls, and include a permutation null to account for layer/pool search. Also examine encoder native prediction/reconstruction mechanisms without training extra readouts.
Limitation
The authors report that for kinetic-energy creation, “fitted probes alone do not establish whether the models distinguish physically impossible energy creation from other collision outcomes,” motivating additional tests using the models’ own prediction/reconstruction.
Abstract (from arXiv)
Video encoder models are increasingly used as world models for physical AI, where a policy plans by rolling out the latent, so what physics the latent holds bounds what the agent can plan over. Moreover, existing interpretability work probes quantities belonging to a single object; the quantities that make physics relational, such as momentum transfer, restitution, and friction, are defined between bodies and have not been examined. We probe relational physics attributes in two frozen self-supervised video encoders (V-JEPA2 and VideoMAE) under two-body settings and test if they transfer across environments. Our early findings are mixed. Both encoders separate elastic from inelastic collisions reliably. V-JEPA2's own predictor is less likely to predict a collision that creates kinetic energy compared to one that loses it. Momentum-violation detection appears in only one of the two encoders, a kinetic-energy readout survives a change of environment for V-JEPA2 but not for VideoMAE, and neither model holds a friction coefficient that survives a change of geometry.
Related papers
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-26 UTC
Interpreting Physics in Video Encoders using Multi-Object Interaction Videos
Preprint posted: 2026-10-22 · Original record ↗
- arxiv
- 2610.31983
Explicit source versions (1)
- v1 · 2026-10-22
- Preprint available
Source date: 2026-10-22 · First detected: 2026-11-02 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video encoder models are increasingly used as world models for physical AI, where a policy plans by rolling out the latent, so what physics the latent holds bounds what the agent can plan over. Moreover, existing interpretability work probes quantities belonging to a single object; the quantities that make physics relational, such as momentum transfer, restitution, and friction, are defined between bodies and have not been examined. We probe relational physics attributes in two frozen self-supervised video encoders (V-JEPA2 and VideoMAE) under two-body settings and test if they transfer across environments. Our early findings are mixed. Both encoders separate elastic from inelastic collisions reliably. V-JEPA2's own predictor is less likely to predict a collision that creates kinetic energy compared to one that loses it. Momentum-violation detection appears in only one of the two encoders, a kinetic-energy readout survives a change of environment for V-JEPA2 but not for VideoMAE, and neither model holds a friction coefficient that survives a change of geometry.
