Latent Dynamics · 2026-10-20
IMPULSE: Push First, Then Predict in V-JEPA 2 Feature Space
Tingan Jin, Chung-Hsien Chou
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
IMPULSE performs three calibration pushes, summarizes their outcomes with a Bayesian contact memory (8 numbers), predicts object pose via a pose pathway, and uses that pose to forecast future V-JEPA 2 feature maps. On a pre-specified held-out confirmation, the memory lowers 1-s feature-forecast NMSE by 11.6–12.8% vs the same predictor without experience (8/8 seeds). A random-Fourier pose head greatly amplifies the gain.
Why it matters
The authors test whether a few active trials improve a predictor whose targets are frozen video-encoder features, and what limits that improvement. Using an information-ladder argument, they isolate how much physics can reach the features only through motion, and they measure how pose pathway design affects the resulting feature forecast gains.
Method
- Calibrate with three short pushes, observe noisy pose tracks, and fit post-contact normal/tangential/angular velocities plus noise estimates.
- Infer an object-specific impulse response using a Bayesian contact memory over (log ρ, e, μ), producing 8 interpretable numbers (posterior mean response + spreads) at two lever arms.
- Use a pose pathway to feed predicted pose into a predictor in frozen V-JEPA 2 feature space; optionally add a random-Fourier pose head to correct predicted features.
Limitation
The task is a simulated planar push with a known object shape, and the renderer shows only geometry; whether the gain carries over to real pushing videos and to 3D scenes whose hidden properties must be inferred from observed interactions remains open.
Abstract (from arXiv)
Latent world models such as V-JEPA 2 forecast in the feature space of a video encoder, but one look at an object does not reveal how hard it is to rotate, how much it bounces or how much it slides. People find out by trying: a few pokes reveal how an object responds. IMPULSE, named after the impulse response it infers, gives a predictor in frozen V-JEPA 2 feature space the same ability. A Bayesian contact memory summarizes three short calibration pushes, a predicted object pose carries it into the predictor, and a lightweight random-Fourier pose head lets the predictor use the pose. On 512 fresh objects, in a pre-specified held-out confirmation with models sealed before the data existed, the memory lowers the 1-second feature-forecast error by 11.6-12.8% relative to the same predictor without experience, for all 8 training seeds, with a 95% seed-and-object interval excluding zero. The gain also holds for 128 objects with out-of-range physics (11.7-14.0%), and the full extension beats a plain predictor by 15.5-16.6%. An information-ladder argument, exact for ideal predictors, shows that physics reaches the features only through motion, so we measure how much the pathway lets pose help. On fresh objects, the Fourier head roughly triples the gain of a perfect memory over plain pose tokens (2.8-3.2 times). Raw push measurements fed through the same pathway give a similar gain, so the Bayesian summary is not required; it condenses the trials into eight interpretable numbers. The gain is absent at contact onset and grows afterwards.
Related papers
- Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- SaiVLA-JEPA: Evaluating Future Latent Prediction in JEPA-Conditioned VLA Control
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Tracking Is Not Permanence: What Video World Models Keep of a Hidden Object
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-20 UTC
IMPULSE: Push First, Then Predict in V-JEPA 2 Feature Space
Preprint posted: 2026-10-12 · Original record ↗
- arxiv
- 2610.23968
Explicit source versions (1)
- v1 · 2026-10-12
- Preprint available
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Latent world models such as V-JEPA 2 forecast in the feature space of a video encoder, but one look at an object does not reveal how hard it is to rotate, how much it bounces or how much it slides. People find out by trying: a few pokes reveal how an object responds. IMPULSE, named after the impulse response it infers, gives a predictor in frozen V-JEPA 2 feature space the same ability. A Bayesian contact memory summarizes three short calibration pushes, a predicted object pose carries it into the predictor, and a lightweight random-Fourier pose head lets the predictor use the pose. On 512 fresh objects, in a pre-specified held-out confirmation with models sealed before the data existed, the memory lowers the 1-second feature-forecast error by 11.6-12.8% relative to the same predictor without experience, for all 8 training seeds, with a 95% seed-and-object interval excluding zero. The gain also holds for 128 objects with out-of-range physics (11.7-14.0%), and the full extension beats a plain predictor by 15.5-16.6%. An information-ladder argument, exact for ideal predictors, shows that physics reaches the features only through motion, so we measure how much the pathway lets pose help. On fresh objects, the Fourier head roughly triples the gain of a perfect memory over plain pose tokens (2.8-3.2x). Raw push measurements fed through the same pathway give a similar gain, so the Bayesian summary is not required; it condenses the trials into eight interpretable numbers. The gain is absent at contact onset and grows afterwards.
