Latent Dynamics · 2026-10-02
STATERA: Hidden Mass Estimation via Zero-Shot Sim-to-Real Kinematics using Frozen Temporal Tubelets
Animesh Varma
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
STATERA estimates hidden center-of-mass (CoM) offsets of opaque, asymmetric rigid bodies from short monocular videos. It adapts a partially-frozen V-JEPA (video JEPA) backbone with temporal tubelet mixing and a lightweight 2.5D decoder to predict per-frame CoM heatmaps/trajectories. The authors introduce the HiddenMass Benchmark (50K MuJoCo + 63 real sequences). They report improved normalized CoM error in simulation and strong physics capture in zero-shot sim-to-real using phase-aware supervision.
Why it matters
The authors address hidden-parameter estimation where surface cues and point tracking fail under self-occlusion. By leveraging frozen temporal representations and predicting CoM via kinematics/heatmaps (rather than direct coordinate regression), STATERA targets inertial dynamics and shows trade-offs between Euclidean error and physics capture during zero-shot sim-to-real transfer.
Method
- Partially-frozen V-JEPA backbone with temporal tubelet mixing: a 16-frame sequence is compressed into temporal tubelets, then a 1D convolution mixes latents across time for non-causal (offline) inference.
- 2.5D multi-task decoder outputs a continuous spatial CoM heatmap (Head A) and an absolute Z-depth regularizer (Head B) from monocular video.
- Training uses curriculum label smoothing with variance decay (phase-agnostic “Sigma” vs phase-aware “Crescent” targets) and temperature-scaled soft-argmax for continuous sub-pixel coordinate extraction.
Limitation
The authors note that STATERA-50K-Crescent can exhibit a monocular vector overshoot artifact that marginally increases absolute Euclidean error compared to a static geometric centroid.
Abstract (from arXiv)
Vision models pretrained for frame-level appearance often struggle to infer hidden physical properties from motion. We study center-of-mass (CoM) localization for opaque, asymmetric rigid bodies from short monocular videos, where surface cues and point tracking are unreliable under self-occlusion. We propose STATERA, which adapts a pretrained video backbone (V-JEPA) with mostly frozen weights and a lightweight temporal tubelet mixer to predict per-frame CoM heatmaps and trajectories. To support this task, we introduce the HiddenMass Benchmark, comprising 50K MuJoCo trajectories and a 63-sequence real-world test set with physically calibrated CoM ground truth. In simulation, STATERA-50K-Sigma improves normalized CoM error from 41.7% (DINOv2) to 25.2%. In zero-shot sim-to-real transfer, we observe a fundamental trade-off in supervision: phase-aware targets can induce bimodal predictions, while phase-agnostic targets can collapse toward statistically safe centroids. Nevertheless, our phase-aware STATERA-50K-Crescent is the only evaluated method that demonstrates consistent movement toward the true hidden offset. While this leads to a monocular vector overshoot artifact that marginally increases absolute Euclidean error compared to a static geometric centroid, it improves physics capture from 2.6% to 41.0%. These results suggest that frozen temporal representations can better separate inertial dynamics from visual geometry for hidden-parameter estimation.
Related papers
- MotionCore: A Latent Neural Simulator for Video Generation with Rigid-Body Dynamics
- Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models
- PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
- Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
- VITAL: From Visual Motion to Precise Action
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-05-21 · Indexed by Pub-AI: 2026-10-02 UTC
STATERA: Hidden Mass Estimation via Zero-Shot Sim-to-Real Kinematics using Frozen Temporal Tubelets
Preprint posted: 2026-05-21 · Original record ↗
- arxiv
- 2610.00003
Explicit source versions (1)
- v1 · 2026-05-21
- Preprint available
Source date: 2026-05-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision models pretrained for frame-level appearance often struggle to infer hidden physical properties from motion. We study center-of-mass (CoM) localization for opaque, asymmetric rigid bodies from short monocular videos, where surface cues and point tracking are unreliable under self-occlusion. We propose STATERA, which adapts a pretrained video backbone (V-JEPA) with mostly frozen weights and a lightweight temporal tubelet mixer to predict per-frame CoM heatmaps and trajectories. To support this task, we introduce the HiddenMass Benchmark, comprising 50K MuJoCo trajectories and a 63-sequence real-world test set with physically calibrated CoM ground truth. In simulation, STATERA-50K-Sigma improves normalized CoM error from 41.7% (DINOv2) to 25.2%. In zero-shot sim-to-real transfer, we observe a fundamental trade-off in supervision: phase-aware targets can induce bimodal predictions, while phase-agnostic targets can collapse toward statistically safe centroids. Nevertheless, our phase-aware STATERA-50K-Crescent is the only evaluated method that demonstrates consistent movement toward the true hidden offset. While this leads to a monocular vector overshoot artifact that marginally increases absolute Euclidean error compared to a static geometric centroid, it improves physics capture from 2.6% to 41.0%. These results suggest that frozen temporal representations can better separate inertial dynamics from visual geometry for hidden-parameter estimation.
