JEPA · 2026-10-23
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Hossein Resani, Javen Qinfeng Shi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Inside-JEPA builds relational channels for ordered entity pairs into a JEPA latent state. The authors show that adding an inverse-dynamics loss—recovering actions from changes in relational channels—grounds these channels without relation labels. On Something-Something V2, prediction alone matches random; inverse supervision improves relational AUROC and supports relational action-conditioned forecasting and one-step goal-based action selection.
Why it matters
JEPAs learn latent dynamics for planning but their states are opaque. The paper tests whether relational structure can emerge inside a world-model latent without relation labels, and which training signals make it meaningful for control. It provides evidence that inverse-dynamics supervision can align relational channels with action-relevant changes in real-video relations and enables relational goal-conditioned planning readouts.
Method
- Inside-JEPA augments JEPA latents with K soft relational channels per ordered entity pair, concatenated with entity latents into the predictor state.
- The authors add an inverse-dynamics loss: an inverse head must recover the action from relational channels before vs. after it, without using relation labels.
- They compare five training variants (different inverse-head inputs/where gradients enter, shuffled targets, label-free losses, and label-fraction controls) and evaluate using one-to-one matched relational AUROC and planning/forecasting probes.
Limitation
The authors state that their matched relational AUROC metric is strict—one channel per relation—and therefore understates a relation spread over several channels (Sec. 4: “it therefore understates a relation spread over several channels”). They also note that the robot result “rests on the given boxes” and “how much it depends on them is untested.”
Abstract (from arXiv)
Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.
Related papers
- Exploring Temporal Context-Aware Physical Grounding in Latent World Models
- World Model Interpretability: From Decoding Representations to Understanding Latent Worlds
- Juno: Taming Predictive Latents for Vision-Language-Action Models
- SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Preprint posted: 2026-10-22 · Original record ↗
- arxiv
- 2610.31194
Explicit source versions (1)
- v1 · 2026-10-22
- Preprint available
Source date: 2026-10-22 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.