Latent Dynamics · 2026-10-21
Exploring Temporal Context-Aware Physical Grounding in Latent World Models
Yilei Man, Yang Chen, Jingcai Guo
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose 𝒢-JEPA, a latent JEPA world model that adds temporal context-aware physical grounding and combines the learned model with prior-guided action planning. They use a shared causal S4D-based temporal module with physical state supervision during training, and during planning mix learned latent-conditioned diffusion action proposals with Gaussian CEM refinement. They report improved planning performance on manipulation tasks, including increasing goal-distance settings.
Why it matters
Latent JEPA world models can lack explicit temporal aggregation in physical grounding and planning can rely too heavily on generic sampling. The paper targets both issues by (1) adding a temporal physical grounding module supervised by physical states and (2) incorporating an offline-learned action prior into online refinement, aiming to improve action planning for control tasks.
Method
- Introduce temporal physical grounding for JEPA using a shared causal S4D module that aggregates temporal context in both observation and prediction sequences, supervised by physical state targets.
- Add nuisance-invariant representation learning via photometric augmentation and an invariance objective using cosine similarity with stop-gradient.
- Use prior-guided planning: learn a latent-conditioned generative action prior (instantiated as a diffusion policy), mix its proposals with Gaussian CEM, rank candidates via latent goal-matching costs from a frozen world model, and update a shared elite set.
Limitation
The temporal physical grounding module is used only during training (not during planning).
Abstract (from arXiv)
Learning latent dynamics is fundamental to building action-conditioned world models for planning. Among existing families, Joint Embedding Predictive Architectures (JEPAs) offer a natural framework by modeling transitions directly in latent space rather than at the pixel level. However, two issues remain in JEPA-based world models: (i) physical grounding lacks explicit aggregation of temporal context and (ii) action planning always overly relies on generic sampling distributions. To this end, we propose $\mathcal{G}$-JEPA, which incorporates temporal context into physical grounding for latent world models and couples the learned models with prior-guided action planning. Specifically, during learning, we incorporate a shared causal temporal module that aggregates temporal context within observation and prediction sequences under physical state supervision, grounding both in task-relevant physical states. When planning, we further introduce action inference through stepwise conditioning of a generative action distribution on the current and goal latent representations together with their evolving discrepancy, yielding anchored proposals that guide model-based online refinement. Together, physical state supervision guides temporal representation learning, while the action prior guides search through the learned world model. Experiments demonstrate improved planning performance on manipulation tasks, including tasks with increasing temporal separation between initial and goal observations. Code is publicly available at https://github.com/myl11045/G-JEPA.
Related papers
- JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
- Time-JEPA: Grounding Video World Models with Physical Time Series
- H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
Exploring Temporal Context-Aware Physical Grounding in Latent World Models
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28483
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Learning latent dynamics is fundamental to building action-conditioned world models for planning. Among existing families, Joint Embedding Predictive Architectures (JEPAs) offer a natural framework by modeling transitions directly in latent space rather than at the pixel level. However, two issues remain in JEPA-based world models: (i) physical grounding lacks explicit aggregation of temporal context and (ii) action planning always overly relies on generic sampling distributions. To this end, we propose $\mathcal{G}$-JEPA, which incorporates temporal context into physical grounding for latent world models and couples the learned models with prior-guided action planning. Specifically, during learning, we incorporate a shared causal temporal module that aggregates temporal context within observation and prediction sequences under physical state supervision, grounding both in task-relevant physical states. When planning, we further introduce action inference through stepwise conditioning of a generative action distribution on the current and goal latent representations together with their evolving discrepancy, yielding anchored proposals that guide model-based online refinement. Together, physical state supervision guides temporal representation learning, while the action prior guides search through the learned world model. Experiments demonstrate improved planning performance on manipulation tasks, including tasks with increasing temporal separation between initial and goal observations. Code is publicly available at https://github.com/myl11045/G-JEPA.