Video World Models · 2026-10-05
Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models
Jonas Kneifl, Jakub Skalski, Bart{\l}omiej Twardowski, Kamil Deja
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors probe internal representations of video Diffusion Transformers (DiTs) using simulator-derived physical quantities (kinematics and rigid-body dynamics). They find that physical quantities are linearly decodable from DiT activations with high accuracy early in denoising, outperforming probes on the model’s own noised latents. The information is sharply localized to tokens near objects (ROI) yet aggregates globally over time, and probing directions can steer generation.
Why it matters
If DiT activations make physical quantities linearly readable—and can be injected to change motion—then video diffusion models contain usable internal representations of physics, not just scene-dependent correlations. This directly informs whether such models can serve as effective world models and whether their internal states can support controllable physical reasoning.
Method
- Use a physics simulator (Genesis) to generate videos with exact ground-truth physical observables; encode videos, forward-diffuse to noise levels, and record frozen DiT layer activations.
- Train ridge regression linear probes (with standardized features) on the DiT residual stream, comparing probe performance from ROI/global pooled activations versus probes on the model’s own noised latents and conditioning.
- Test spatial/temporal localization and temporal aggregation via token/phase selection, then evaluate transfer to out-of-distribution scene variations with recalibration, and perform activation steering using full-resolution probing directions.
Limitation
The paper notes that extrapolation transfer may fail due to gain/offset calibration mismatch (R2 being unbounded below), as well as that global readouts do not survive scene changes as well as ROI readouts.
Abstract (from arXiv)
Video generation models produce strikingly realistic sequences and are increasingly proposed as world models, yet recent benchmarks reveal pronounced deficits in their physical reasoning. This raises the question of whether these models internalize physical principles or merely reproduce familiar motion patterns. We address this by probing internal representations of video Diffusion Transformers (DiTs) for simulator-derived ground-truth physical quantities spanning kinematic motion and rigid-body dynamics under gravity and contact. We find that these quantities are linearly decodable with high accuracy early in the denoising process, substantially outperforming a baseline decoded directly from the model's own noised latents, indicating that the relevant physical information is actively constructed during denoising rather than already present in the input. Additionally, we show that activations at on-object tokens carry the relevant physical information and that quantities defined over multiple frames are readable from single latent frames. Hence, information is sharply localized within the token sequence and is computed globally but stored locally. The probes further show partial extrapolation, transferring to scene variations and object configurations outside their training regime, so what they read is not simply a correlate of the scenes they were fit on. When fitted directly in the full-resolution activation space, the probing directions can serve as steering vectors to change the model's output.
Related papers
- World Models' Last Exam in Physics
- World Embedding Benchmark
- PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
- PhysTrace: Interpretable Temporal Evaluation of Physical Consistency in Video Generative Models
- World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.03154
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video generation models produce strikingly realistic sequences and are increasingly proposed as world models, yet recent benchmarks reveal pronounced deficits in their physical reasoning. This raises the question of whether these models internalize physical principles or merely reproduce familiar motion patterns. We address this by probing internal representations of video Diffusion Transformers (DiTs) for simulator-derived ground-truth physical quantities spanning kinematic motion and rigid-body dynamics under gravity and contact. We find that these quantities are linearly decodable with high accuracy early in the denoising process, substantially outperforming a baseline decoded directly from the model's own noised latents, indicating that the relevant physical information is actively constructed during denoising rather than already present in the input. Additionally, we show that activations at on-object tokens carry the relevant physical information and that quantities defined over multiple frames are readable from single latent frames. Hence, information is sharply localized within the token sequence and is computed globally but stored locally. The probes further show partial extrapolation, transferring to scene variations and object configurations outside their training regime, so what they read is not simply a correlate of the scenes they were fit on. When fitted directly in the full-resolution activation space, the probing directions can serve as steering vectors to change the model's output.