JEPA · 2026-10-28
SuperJEPA: Pretraining with a Supervision-Augmented Joint-Embedding Predictive Architecture for High Energy Physics
Dylan Sheldon Rankin, Ho Fung Tsoi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SuperJEPA introduces a joint-embedding predictive architecture (JEPA) augmented with a supervised classification objective for high energy physics. The authors report improved downstream jet-tagging performance over supervised pretraining alone across multiple tasks, model sizes, and pretraining dataset sizes, with gains persisting for both frozen and fine-tuned encoders.
Why it matters
The paper asks whether self-supervised objectives add value when large, high-fidelity labeled simulation is already available in collider physics. The authors motivate predictive self-supervision for future particle-physics foundation models, showing improvements over supervised pretraining alone across jet-tagging tasks and different training/fine-tuning settings.
Method
- Preprocess each jet by reclustering constituents into subjets (C/A with R=2.0, up to Kmax=8) and define context/target regions via a target transverse-momentum fraction f* in [0.10, 0.35].
- Use a teacher-free LeJEPA-style setup with a single encoder applied to both full and masked-context views; avoid EMA target encoders/stop-gradient, with gradients flowing into both branches.
- Train with a prediction loss via optimal-transport matching between predicted and target sets, plus SIGReg for diversity and a cross-entropy loss using a [CLS] token; evaluate with frozen or fine-tuned encoders.
Limitation
The authors state: “While we use pretraining datasets with up to 20M jets, we are unable to test on datasets into the 100Ms and 1Bs. Future studies are necessary to understand if our conclusions continue to hold with these even larger datasets.”
Abstract (from arXiv)
Foundation models often obtain powerful results through very large unlabeled training datasets that dwarf available labeled datasets in size, but high energy physics already provides large samples of high-fidelity labeled simulation. We therefore ask whether self-supervised objectives improve over supervised pretraining alone when large labeled samples already exist. We introduce SuperJEPA, a joint-embedding predictive architec- ture augmented with a supervised objective, and find improved downstream jet-tagging performance across multiple tasks, model sizes, and pretraining dataset sizes. The gains persist for both frozen and fine-tuned encoders, motivating predictive self-supervision in future large-scale particle-physics foundation models.
Related papers
- Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- World Embedding Benchmark
- Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-26 · Indexed by Pub-AI: 2026-10-28 UTC
SuperJEPA: Pretraining with a Supervision-Augmented Joint-Embedding Predictive Architecture for High Energy Physics
Preprint posted: 2026-10-26 · Original record ↗
- arxiv
- 2610.36121
Explicit source versions (1)
- v1 · 2026-10-26
- Preprint available
Source date: 2026-10-26 · First detected: 2026-11-01 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Foundation models often obtain powerful results through very large unlabeled training datasets that dwarf available labeled datasets in size, but high energy physics already provides large samples of high-fidelity labeled simulation. We therefore ask whether self-supervised objectives improve over supervised pretraining alone when large labeled samples already exist. We introduce SuperJEPA, a joint-embedding predictive architec- ture augmented with a supervised objective, and find improved downstream jet-tagging performance across multiple tasks, model sizes, and pretraining dataset sizes. The gains persist for both frozen and fine-tuned encoders, motivating predictive self-supervision in future large-scale particle-physics foundation models.