JEPA · 2026-10-22
Are Joint-Embedding Predictive Objectives Effective for Longitudinal EHRs?
Ashvin Gupta, Brendan Delaney, Alessandra Russo
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors adapt Joint-Embedding Predictive Architectures (JEPAs) to longitudinal EHR sequences (MIMIC-IV) using three target constructions (Span, Future, Random) and compare against masked language modelling (MLM). Although all JEPA variants train stably without representational collapse, the authors report that all three consistently underperform standard MLM on downstream clinical prediction tasks.
Why it matters
This work tests whether stable latent-space prediction (JEPA) can replace token-level reconstruction (MLM) for longitudinal EHR representation learning. The authors find stable non-collapsed training is not enough: JEPA representations are weaker for downstream AUROC/AUPRC, and target-size changes do not recover performance. It highlights the challenge of defining clinically meaningful latent targets for EHR.
Method
- Train transformer encoders on MIMIC-IV with an EMA target encoder (stop-gradient) and a predictor matched via mean squared error between predicted and target latent representations.
- Evaluate JEPA objectives with three target constructions: contiguous span masking (Span JEPA), time-window future prediction (Future JEPA), and random event masking (Random JEPA), keeping transformer capacity fixed.
- Use frozen-encoder linear probes (mean pooling) to assess representation quality on ICU/hospital mortality/readmission tasks (AUROC/AUPRC), and study stability using effective rank and embedding feature variance; also analyze which target properties predict latent-prediction difficulty.
Abstract (from arXiv)
Masked language modelling (MLM) is the dominant pretraining objective for transformer encoders applied to longitudinal electronic health records (EHRs). However, reconstructing individual events may overemphasise noisy or weakly informative observations and does not explicitly encourage learning higher-level patient states. Joint-Embedding Predictive Architectures (JEPAs) instead learn by predicting representations in latent space and have shown strong performance in a variety of domains. In this paper, we investigate to what extent JEPA can be adapted to EHR sequence representation. Our empirical investigation using MIMIC-IV shows that although all three of our JEPA formulations train stably without representational collapse, they consistently underperform standard MLM. We find that target size, not target heterogeneity, increases prediction error, yet reducing the target size does not recover downstream performance. This suggests that stable latent prediction alone is insufficient for competitive EHR representation learning and highlights the challenge of defining clinically meaningful latent targets for EHR.
Related papers
- ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models
- Does Joint-Embedding Predictive Architecture Pretraining Help Time Series Forecasting?
- Forecasting Patient Trajectories from the Electronic Health Records with Action-Conditioned World Models
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
- DES-JEPA: Learning a Shared Vocabulary for High-Dimensional Discrete Event Sequences
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC
Are Joint-Embedding Predictive Objectives Effective for Longitudinal EHRs?
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.29528
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Masked language modelling (MLM) is the dominant pretraining objective for transformer encoders applied to longitudinal electronic health records (EHRs). However, reconstructing individual events may overemphasise noisy or weakly informative observations and does not explicitly encourage learning higher-level patient states. Joint-Embedding Predictive Architectures (JEPAs) instead learn by predicting representations in latent space and have shown strong performance in a variety of domains. In this paper, we investigate to what extent JEPA can be adapted to EHR sequence representation. Our empirical investigation using MIMIC-IV shows that although all three of our JEPA formulations train stably without representational collapse, they consistently underperform standard MLM. We find that target size, not target heterogeneity, increases prediction error, yet reducing the target size does not recover downstream performance. This suggests that stable latent prediction alone is insufficient for competitive EHR representation learning and highlights the challenge of defining clinically meaningful latent targets for EHR.