Driving · 2026-10-26

Interpreting Physics in Video Encoders using Multi-Object Interaction Videos

Ayushi Mehrotra, Sai Vidyaranya Nuthalapati

arXiv:2610.31983PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors probe whether two frozen self-supervised video encoders (V-JEPA2 and VideoMAE) encode relational physics in two-object interactions (momentum conservation, elasticity, kinetic-energy changes) and in cube–floor friction. They find elastic vs inelastic collisions are reliably separated in both, but momentum-violation and kinetic-energy transfer across environments are mixed, and friction-coefficient readouts do not survive geometry changes.

Why it matters

If a physical AI policy plans by rolling out a video-encoder latent, then what relational quantities (e.g., momentum transfer, restitution, friction) the latent represents determines what the agent can plan. The paper directly measures relational-physics attributes inside frozen encoders and tests whether such readouts transfer across scene changes, helping distinguish genuine physics information from visual correlates.

Method

  • Use two frozen self-supervised video encoders (VideoMAE-base and V-JEPA2-ViT-L) and probe pooled activations with ridge-regression probes (continuous: out-of-fold R2; binary: out-of-fold AUC).
  • Probe relational quantities for two-body collisions (momentum conservation, elasticity, kinetic-energy changes) and friction for cube–floor settings; test transfer by fitting probes in one scenario and applying unchanged in another.
  • Use matched observers with access to visible kinematic scalars as controls, and include a permutation null to account for layer/pool search. Also examine encoder native prediction/reconstruction mechanisms without training extra readouts.

Limitation

The authors report that for kinetic-energy creation, “fitted probes alone do not establish whether the models distinguish physically impossible energy creation from other collision outcomes,” motivating additional tests using the models’ own prediction/reconstruction.

Abstract (from arXiv)

Video encoder models are increasingly used as world models for physical AI, where a policy plans by rolling out the latent, so what physics the latent holds bounds what the agent can plan over. Moreover, existing interpretability work probes quantities belonging to a single object; the quantities that make physics relational, such as momentum transfer, restitution, and friction, are defined between bodies and have not been examined. We probe relational physics attributes in two frozen self-supervised video encoders (V-JEPA2 and VideoMAE) under two-body settings and test if they transfer across environments. Our early findings are mixed. Both encoders separate elastic from inelastic collisions reliably. V-JEPA2's own predictor is less likely to predict a collision that creates kinetic energy compared to one that loses it. Momentum-violation detection appears in only one of the two encoders, a kinetic-energy readout survives a change of environment for V-JEPA2 but not for VideoMAE, and neither model holds a friction coefficient that survives a change of geometry.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-26 UTC

preprint

Interpreting Physics in Video Encoders using Multi-Object Interaction Videos

Ayushi Mehrotra, Sai Vidyaranya Nuthalapati

Preprint posted: 2026-10-22 · Original record ↗

arxiv
2610.31983
Explicit source versions (1)
  • v1 · 2026-10-22
  1. Preprint available

    Source date: 2026-10-22 · First detected: 2026-11-02 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Video encoder models are increasingly used as world models for physical AI, where a policy plans by rolling out the latent, so what physics the latent holds bounds what the agent can plan over. Moreover, existing interpretability work probes quantities belonging to a single object; the quantities that make physics relational, such as momentum transfer, restitution, and friction, are defined between bodies and have not been examined. We probe relational physics attributes in two frozen self-supervised video encoders (V-JEPA2 and VideoMAE) under two-body settings and test if they transfer across environments. Our early findings are mixed. Both encoders separate elastic from inelastic collisions reliably. V-JEPA2's own predictor is less likely to predict a collision that creates kinetic energy compared to one that loses it. Momentum-violation detection appears in only one of the two encoders, a kinetic-energy readout survives a change of environment for V-JEPA2 but not for VideoMAE, and neither model holds a friction coefficient that survives a change of geometry.