Latent Dynamics · 2026-10-02

STATERA: Hidden Mass Estimation via Zero-Shot Sim-to-Real Kinematics using Frozen Temporal Tubelets

Animesh Varma

arXiv:2610.00003PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

STATERA estimates hidden center-of-mass (CoM) offsets of opaque, asymmetric rigid bodies from short monocular videos. It adapts a partially-frozen V-JEPA (video JEPA) backbone with temporal tubelet mixing and a lightweight 2.5D decoder to predict per-frame CoM heatmaps/trajectories. The authors introduce the HiddenMass Benchmark (50K MuJoCo + 63 real sequences). They report improved normalized CoM error in simulation and strong physics capture in zero-shot sim-to-real using phase-aware supervision.

Why it matters

The authors address hidden-parameter estimation where surface cues and point tracking fail under self-occlusion. By leveraging frozen temporal representations and predicting CoM via kinematics/heatmaps (rather than direct coordinate regression), STATERA targets inertial dynamics and shows trade-offs between Euclidean error and physics capture during zero-shot sim-to-real transfer.

Method

  • Partially-frozen V-JEPA backbone with temporal tubelet mixing: a 16-frame sequence is compressed into temporal tubelets, then a 1D convolution mixes latents across time for non-causal (offline) inference.
  • 2.5D multi-task decoder outputs a continuous spatial CoM heatmap (Head A) and an absolute Z-depth regularizer (Head B) from monocular video.
  • Training uses curriculum label smoothing with variance decay (phase-agnostic “Sigma” vs phase-aware “Crescent” targets) and temperature-scaled soft-argmax for continuous sub-pixel coordinate extraction.

Limitation

The authors note that STATERA-50K-Crescent can exhibit a monocular vector overshoot artifact that marginally increases absolute Euclidean error compared to a static geometric centroid.

Abstract (from arXiv)

Vision models pretrained for frame-level appearance often struggle to infer hidden physical properties from motion. We study center-of-mass (CoM) localization for opaque, asymmetric rigid bodies from short monocular videos, where surface cues and point tracking are unreliable under self-occlusion. We propose STATERA, which adapts a pretrained video backbone (V-JEPA) with mostly frozen weights and a lightweight temporal tubelet mixer to predict per-frame CoM heatmaps and trajectories. To support this task, we introduce the HiddenMass Benchmark, comprising 50K MuJoCo trajectories and a 63-sequence real-world test set with physically calibrated CoM ground truth. In simulation, STATERA-50K-Sigma improves normalized CoM error from 41.7% (DINOv2) to 25.2%. In zero-shot sim-to-real transfer, we observe a fundamental trade-off in supervision: phase-aware targets can induce bimodal predictions, while phase-agnostic targets can collapse toward statistically safe centroids. Nevertheless, our phase-aware STATERA-50K-Crescent is the only evaluated method that demonstrates consistent movement toward the true hidden offset. While this leads to a monocular vector overshoot artifact that marginally increases absolute Euclidean error compared to a static geometric centroid, it improves physics capture from 2.6% to 41.0%. These results suggest that frozen temporal representations can better separate inertial dynamics from visual geometry for hidden-parameter estimation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-05-21 · Indexed by Pub-AI: 2026-10-02 UTC

preprint

STATERA: Hidden Mass Estimation via Zero-Shot Sim-to-Real Kinematics using Frozen Temporal Tubelets

Animesh Varma

Preprint posted: 2026-05-21 · Original record ↗

arxiv
2610.00003
Explicit source versions (1)
  • v1 · 2026-05-21
  1. Preprint available

    Source date: 2026-05-21 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Vision models pretrained for frame-level appearance often struggle to infer hidden physical properties from motion. We study center-of-mass (CoM) localization for opaque, asymmetric rigid bodies from short monocular videos, where surface cues and point tracking are unreliable under self-occlusion. We propose STATERA, which adapts a pretrained video backbone (V-JEPA) with mostly frozen weights and a lightweight temporal tubelet mixer to predict per-frame CoM heatmaps and trajectories. To support this task, we introduce the HiddenMass Benchmark, comprising 50K MuJoCo trajectories and a 63-sequence real-world test set with physically calibrated CoM ground truth. In simulation, STATERA-50K-Sigma improves normalized CoM error from 41.7% (DINOv2) to 25.2%. In zero-shot sim-to-real transfer, we observe a fundamental trade-off in supervision: phase-aware targets can induce bimodal predictions, while phase-agnostic targets can collapse toward statistically safe centroids. Nevertheless, our phase-aware STATERA-50K-Crescent is the only evaluated method that demonstrates consistent movement toward the true hidden offset. While this leads to a monocular vector overshoot artifact that marginally increases absolute Euclidean error compared to a static geometric centroid, it improves physics capture from 2.6% to 41.0%. These results suggest that frozen temporal representations can better separate inertial dynamics from visual geometry for hidden-parameter estimation.