JEPA · 2026-10-28

SWIM: Learning What Matters from Behaviors for JEPA World Models

Jiaming Hu, Shi Bo, Yan Zheng, Tian Wang, Florian Dubost, Alejandro Mottini, Junze Liu, Arvind Srinivasan, Kai Zhong, Kun Qian, Sharon Gao, Vianne Gao, Houyu Zhang, Qingjun Cui

arXiv:2610.36662PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SWIM learns which simulator state coordinates matter for JEPA-style planning from offline behavior. It trains a gated inverse-dynamics model to predict logged actions from current states and observed future displacements, using it to define a learned weighted state metric for state-supervised training. The inverse-dynamics model is only used during training, and SWIM improves planning success over uniform state weighting across five simulated robotic manipulation tasks.

Why it matters

State-supervised JEPA calibration (e.g., SCALE) depends on how state coordinates are selected and weighted; uniform weighting can overcount redundant variation and can let irrelevant coordinates obscure task-relevant differences. SWIM replaces hand or uniform state weighting with weights learned from offline trajectories using an inverse-dynamics signal, then transfers the learned metric to calibrate latent geometry without adding planning-time computation.

Method

  • Stage 1: learn continuous, gated per-coordinate state weights by training an inverse-dynamics predictor to model recorded actions from current states and displacement, with a discrimination term and an ℓ1 gate penalty.
  • Stage 2: freeze the learned gates and replace SCALE’s uniformly weighted state-distance targets with weighted squared state distances defined by the learned metric; train the JEPA encoder accordingly.
  • Deployment: plan from images using the learned latent cost, with the inverse-dynamics components discarded (no simulator states or additional computation at inference).
Abstract (from arXiv)

Joint-embedding predictive architectures (JEPAs) enable world models to plan from images by predicting future embeddings. However, their visual prediction objectives do not explicitly incorporate task-relevant state information, so the physical state that determines task progress may not be reliably captured in the latent space. Supervising world models with simulator state can make planning more informative, but it requires deciding which state features matter. Features are typically selected by hand or weighted uniformly, and a uniform weighting gives every standardized state feature equal influence, so redundant coordinates can overcount the same variation, while irrelevant ones can obscure task-relevant differences. To address this limitation, we propose SWIM (State Weighting with Inverse-Dynamics Model), which learns state weights from offline trajectories by predicting recorded actions from current states and observed future displacements. The learned weights quantify each feature's contribution to action prediction and define a weighted state metric for state-supervised training, which we demonstrate by aligning latent and state distances. The inverse-dynamics model is used only during training, so the resulting world model plans from images with no additional planning-time computation. Across five simulated robotic manipulation tasks, SWIM consistently improves planning success over uniform state weighting.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC

preprint

SWIM: Learning What Matters from Behaviors for JEPA World Models

Jiaming Hu, Shi Bo, Yan Zheng, Tian Wang, Florian Dubost, Alejandro Mottini, Junze Liu, Arvind Srinivasan, Kai Zhong, Kun Qian, Sharon Gao, Vianne Gao, Houyu Zhang, Qingjun Cui

Preprint posted: 2026-10-27 · Original record ↗

arxiv
2610.36662
Explicit source versions (1)
  • v1 · 2026-10-27
  1. Preprint available

    Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Joint-embedding predictive architectures (JEPAs) enable world models to plan from images by predicting future embeddings. However, their visual prediction objectives do not explicitly incorporate task-relevant state information, so the physical state that determines task progress may not be reliably captured in the latent space. Supervising world models with simulator state can make planning more informative, but it requires deciding which state features matter. Features are typically selected by hand or weighted uniformly, and a uniform weighting gives every standardized state feature equal influence, so redundant coordinates can overcount the same variation, while irrelevant ones can obscure task-relevant differences. To address this limitation, we propose SWIM (State Weighting with Inverse-Dynamics Model), which learns state weights from offline trajectories by predicting recorded actions from current states and observed future displacements. The learned weights quantify each feature's contribution to action prediction and define a weighted state metric for state-supervised training, which we demonstrate by aligning latent and state distances. The inverse-dynamics model is used only during training, so the resulting world model plans from images with no additional planning-time computation. Across five simulated robotic manipulation tasks, SWIM consistently improves planning success over uniform state weighting.