Driving · 2026-10-21
Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
Peisong Wen, Qianqian Xu, Minye Lei, Siran Dai, Yang Liu, Qingming Huang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Slot-JEPA proposes a slot-and-mask interface for latent joint-embedding predictive world models. A pretrained slot decomposition module factorizes dense patch representations into stable semantic slots and time-varying spatial masks, then is frozen for downstream prediction and control. A Transition Memory Predictor (TMP) retrieves compact, slot-aligned transition context from a memory bank. Unbalanced Optimal Transport (UOT) supervises mask prediction and provides a displacement-aware goal evaluation signal for MPC.
Why it matters
Dense JEPA patch representations require downstream predictors to recover implicit spatial organization and temporal correspondence. Slot-JEPA exposes object-structured organization via reusable slot/mask states, supports compact memory storage and retrieval using slot correspondence, and replaces pointwise distance losses with UOT that better reflects spatial progress when predicted and goal supports are disjoint.
Method
- Pretrain a slot decomposition module to factor dense patch representations into stable semantic slots and time-varying spatial masks; freeze this module and the visual encoder for downstream tasks.
- Train TMP to predict next-frame masks using retrieved context transitions stored and sampled from transition memory; separate slot-specific dynamics from shared cross-slot interaction dynamics.
- Use Unbalanced Optimal Transport (UOT) to supervise mask prediction and to evaluate candidate plans by transporting mass between aligned predicted and goal masks in MPC.
Abstract (from arXiv)
As a prototypical latent world model, Joint-Embedding Predictive Architectures (JEPAs) provide strong semantic representations. However, dense patch representations expose a redundant interface to downstream predictors, which must implicitly infer spatial organization and temporal correspondence, presenting three challenges. First, object information is distributed across patches, forcing each downstream predictor to redundantly recover object structure that could otherwise be reused across tasks before modeling interactions. Second, the lack of explicit entity correspondence across time makes historical transitions expensive, restricting the predictor's flexibility in utilizing context. Third, without explicit transport structure, existing JEPA models typically use pointwise l1/l2 distances, which are insensitive to spatial displacement when predicted and goal supports are largely disjoint. We propose the Slot-based Joint-Embedding Predictive Architecture (Slot-JEPA), which decomposes dense representations into stable semantic slots and time-varying spatial masks. Pretrained once and frozen downstream, this decomposition exposes a reusable object-structured interface. Building on it, our Transition Memory Predictor (TMP) stores visual transitions as compact masks and retrieves useful context using statistics of corresponding slots. Its two branches separate slot-specific dynamics from shared cross-slot dynamics. We further use Unbalanced Optimal Transport (UOT) to supervise mask prediction and evaluate candidate plans by transporting mass between aligned predicted and goal masks, providing a signal better aligned with spatial progress. Across seven visual-understanding benchmarks, the slot-reconstructed features remain transferable. Slot-JEPA also improves goal-conditioned planning across six planar environments and four robotic arm planning tasks on the REALM-JEPA benchmark.
Related papers
- Enhancing Latent World Models with Associative Memory
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
- LeWAM: End-to-End World and Action Modeling with JEPAs
- MDN-VJEPA: Multimodal Probabilistic Joint-Embedding World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28074
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · OAI arXiv created; not header datestamp (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
As a prototypical latent world model, Joint-Embedding Predictive Architectures (JEPAs) provide strong semantic representations. However, dense patch representations expose a redundant interface to downstream predictors, which must implicitly infer spatial organization and temporal correspondence, presenting three challenges. First, object information is distributed across patches, forcing each downstream predictor to redundantly recover object structure that could otherwise be reused across tasks before modeling interactions. Second, the lack of explicit entity correspondence across time makes historical transitions expensive, restricting the predictor's flexibility in utilizing context. Third, without explicit transport structure, existing JEPA models typically use pointwise l1/l2 distances, which are insensitive to spatial displacement when predicted and goal supports are largely disjoint. We propose the Slot-based Joint-Embedding Predictive Architecture (Slot-JEPA), which decomposes dense representations into stable semantic slots and time-varying spatial masks. Pretrained once and frozen downstream, this decomposition exposes a reusable object-structured interface. Building on it, our Transition Memory Predictor (TMP) stores visual transitions as compact masks and retrieves useful context using statistics of corresponding slots. Its two branches separate slot-specific dynamics from shared cross-slot dynamics. We further use Unbalanced Optimal Transport (UOT) to supervise mask prediction and evaluate candidate plans by transporting mass between aligned predicted and goal masks, providing a signal better aligned with spatial progress. Across seven visual-understanding benchmarks, the slot-reconstructed features remain transferable. Slot-JEPA also improves goal-conditioned planning across six planar environments and four robotic arm planning tasks on the REALM-JEPA benchmark.