JEPA · 2026-10-30
DES-JEPA: Learning a Shared Vocabulary for High-Dimensional Discrete Event Sequences
Hugo Math, Maximilian Forstenh\"ausler, Christos Anagnostopoulos
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DES-JEPA learns a shared discrete codebook for discrete event sequences with unaligned, domain-specific event vocabularies by training on synthetic stochastic processes. DES-PFN reuses the frozen codebook for in-context next-event prediction (zero-shot, no task-specific fine-tuning). The authors report competitive zero-shot performance on eight real datasets and scaling to 700 distinct event types.
Why it matters
The authors address a limitation of prior foundation inference for discrete event sequences (FIM-PP supports at most 22 event types). By learning a reusable shared vocabulary (codebook) across processes with disjoint event vocabularies, DES-JEPA/DES-PFN aims to transfer discrete temporal dynamics without aligning raw event labels, enabling next-event prediction on high-dimensional event streams.
Method
- DES-JEPA: a JEPA-based joint-embedding predictive architecture with a teacher encoder that quantizes future windows into a shared discrete codebook and a student encoder that predicts latent codes from history.
- DES-PFN: freezes the DES-JEPA codebook and trains an in-context autoregressive next-event predictor over code tokens for zero-shot transfer to unseen datasets.
- Decoding: maps predicted codes back to event types via a context-specific kernel (Nadaraya-Watson estimator) using token–event pairs from the context.
Limitation
Two limitations motivate future work: (i) the codebook capacity is fixed during pretraining, and currently lacks rate-distortion experiments; (ii) evaluation is restricted to one-step-ahead prediction, and long-horizon autoregressive rollouts remain untested.
Abstract (from arXiv)
Temporal foundation models such as Chronos and TimesFM generalize across domains through a shared numerical observation space. In contrast, discrete event sequences (DESs), including operational event logs or patient clinical-event trajectories, have domain-specific event vocabularies whose labels lack a consistent correspondence across domains. Beyond this lack of consistency, the number of distinct events in real-world settings is high enough to challenge both classical temporal point process modeling and current in-context foundation inference models such as FIM-PP. Critically, current foundation models for DESs are limited to a small, fixed number of event types, restricting their applicability to realistic, high-dimensional event streams. To address this, we propose two complementary modules: (i) DES-JEPA, a joint-embedding predictive architecture that learns a shared discrete vocabulary from synthetic stochastic processes with unaligned event vocabularies; and (ii) DES-PFN, a discrete event sequence prior-fitted network that reuses the learned vocabulary for in-context next-event prediction. Without task-specific fine-tuning, this framework achieves competitive zero-shot performance on eight unseen real-world datasets and scales up to 700 distinct event types.
Related papers
- Are Joint-Embedding Predictive Objectives Effective for Longitudinal EHRs?
- Forecasting Patient Trajectories from the Electronic Health Records with Action-Conditioned World Models
- Self-Supervised Latent Action-Conditioned World Modeling for Time Series
- PatientWM: Towards Generalizable Clinical World Models via Unified Patient States
- Pythia: Toward Foundation World Models for Multimodal Time Series
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-10-30 · Indexed by Pub-AI: 2026-10-30 UTC
DES-JEPA: Learning a Shared Vocabulary for High-Dimensional Discrete Event Sequences
First announced by arXiv: 2026-10-30 · Original record ↗
- arxiv
- 2610.38445
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-10-30 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.38445v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Temporal foundation models such as Chronos and TimesFM generalize across domains through a shared numerical observation space. In contrast, discrete event sequences (DESs), including operational event logs or patient clinical-event trajectories, have domain-specific event vocabularies whose labels lack a consistent correspondence across domains. Beyond this lack of consistency, the number of distinct events in real-world settings is high enough to challenge both classical temporal point process modeling and current in-context foundation inference models such as FIM-PP. Critically, current foundation models for DESs are limited to a small, fixed number of event types, restricting their applicability to realistic, high-dimensional event streams. To address this, we propose two complementary modules: (i) DES-JEPA, a joint-embedding predictive architecture that learns a shared discrete vocabulary from synthetic stochastic processes with unaligned event vocabularies; and (ii) DES-PFN, a discrete event sequence prior-fitted network that reuses the learned vocabulary for in-context next-event prediction. Without task-specific fine-tuning, this framework achieves competitive zero-shot performance on eight unseen real-world datasets and scales up to 700 distinct event types.