JEPA · 2026-10-20
MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures
Haowen Bai, Rui Gong$, Meichen Liu, Ming-Xuan Weng, Bihan Wen, Kim-Hui Yap
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
MM-LeJEPA extends LeJEPA to multimodal (audio-visual) joint-embedding predictive architectures by replacing LeJEPA’s isotropic task prior with a multimodal task prior that models independent information and shared content across modalities. Under stated assumptions, it derives a correlated Gaussian target with a single cross-modal correlation hyperparameter ρ, enforced via SIGReg using a fixed whitening transform. Code will be released.
Why it matters
LeJEPA’s isotropic Gaussian target can penalize cross-modal correlation when multimodal tasks read shared content from both modalities. MM-LeJEPA builds a task-prior from independent and shared-content components and derives a correlated Gaussian target that explicitly models cross-modal correlation, while keeping LeJEPA-like simplicity (no reconstruction decoder, EMA teacher, or contrastive negatives). This aims to make multimodal self-supervised joint-embedding pretraining fit the structure of shared-content multimodal tasks.
Method
- Build a multimodal task prior that combines independent-information and shared-content components; derive a joint task-gradient second moment W_AV carrying cross-modal structure.
- Minimize a weighted Fisher-information functional under a fixed total-variance constraint to obtain a correlated Gaussian target N(0, Σ*(ρ)) with a single cross-modal correlation parameter ρ.
- Enforce the target by applying a fixed whitening transform to map Σ*(ρ) to identity, then applying SIGReg to transformed embeddings; prediction is local-to-global mean-squared error.
Abstract (from arXiv)
LeJEPA derives an isotropic Gaussian embedding target by minimizing, under its task assumptions, a Fisher information criterion that controls the task-averaged squared prediction bias. This enables unimodal self-supervised learning without reconstruction decoders, exponential-moving-average teachers, or contrastive negatives. However, its isotropic task prior does not account for the cross-modal structure of multimodal tasks that read shared content. We introduce MM-LeJEPA by extending LeJEPA's task assumptions to multimodal learning. Specifically, we model multimodal tasks as strength-weighted combinations of components that read independent information from the two modalities and components that read the same content from both. Under explicit gradient-moment, correspondence, and symmetry assumptions, we derive a correlated Gaussian target that minimizes a weighted Fisher information functional under a fixed total variance constraint. An analytic transform adapts Sketched Isotropic Gaussian Regularization (SIGReg) to this target, yielding a two-term objective that combines distributional regularization with local-to-global prediction. The extension adds a single scalar hyperparameter, the target cross-modal correlation, and retains the simplicity of LeJEPA. On public audio-visual classification benchmarks, our results are competitive with representative open-source methods. Code will be released.
Related papers
- A Stochastic Latent Variable Generalization of Joint-Embedding Predictive Architectures
- LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
- Asymmetric Multimodal Learning with Decomposed JEPAs
- LEF-JEPA: Denoising-Trained Latent Energy Fields for Spatial Representation Learning
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-17 · Indexed by Pub-AI: 2026-10-20 UTC
MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures
Preprint posted: 2026-10-17 · Original record ↗
- arxiv
- 2610.24681
Explicit source versions (1)
- v1 · 2026-10-17
- Preprint available
Source date: 2026-10-17 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
LeJEPA derives an isotropic Gaussian embedding target by minimizing, under its task assumptions, a Fisher information criterion that controls the task-averaged squared prediction bias. This enables unimodal self-supervised learning without reconstruction decoders, exponential-moving-average teachers, or contrastive negatives. However, its isotropic task prior does not account for the cross-modal structure of multimodal tasks that read shared content. We introduce MM-LeJEPA by extending LeJEPA's task assumptions to multimodal learning. Specifically, we model multimodal tasks as strength-weighted combinations of components that read independent information from the two modalities and components that read the same content from both. Under explicit gradient-moment, correspondence, and symmetry assumptions, we derive a correlated Gaussian target that minimizes a weighted Fisher information functional under a fixed total variance constraint. An analytic transform adapts Sketched Isotropic Gaussian Regularization (SIGReg) to this target, yielding a two-term objective that combines distributional regularization with local-to-global prediction. The extension adds a single scalar hyperparameter, the target cross-modal correlation, and retains the simplicity of LeJEPA. On public audio-visual classification benchmarks, our results are competitive with representative open-source methods. Code will be released.
