JEPA · 2026-10-20

MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures

Haowen Bai, Rui Gong$, Meichen Liu, Ming-Xuan Weng, Bihan Wen, Kim-Hui Yap

arXiv:2610.24681PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

MM-LeJEPA extends LeJEPA to multimodal (audio-visual) joint-embedding predictive architectures by replacing LeJEPA’s isotropic task prior with a multimodal task prior that models independent information and shared content across modalities. Under stated assumptions, it derives a correlated Gaussian target with a single cross-modal correlation hyperparameter ρ, enforced via SIGReg using a fixed whitening transform. Code will be released.

Why it matters

LeJEPA’s isotropic Gaussian target can penalize cross-modal correlation when multimodal tasks read shared content from both modalities. MM-LeJEPA builds a task-prior from independent and shared-content components and derives a correlated Gaussian target that explicitly models cross-modal correlation, while keeping LeJEPA-like simplicity (no reconstruction decoder, EMA teacher, or contrastive negatives). This aims to make multimodal self-supervised joint-embedding pretraining fit the structure of shared-content multimodal tasks.

Method

  • Build a multimodal task prior that combines independent-information and shared-content components; derive a joint task-gradient second moment W_AV carrying cross-modal structure.
  • Minimize a weighted Fisher-information functional under a fixed total-variance constraint to obtain a correlated Gaussian target N(0, Σ*(ρ)) with a single cross-modal correlation parameter ρ.
  • Enforce the target by applying a fixed whitening transform to map Σ*(ρ) to identity, then applying SIGReg to transformed embeddings; prediction is local-to-global mean-squared error.
Abstract (from arXiv)

LeJEPA derives an isotropic Gaussian embedding target by minimizing, under its task assumptions, a Fisher information criterion that controls the task-averaged squared prediction bias. This enables unimodal self-supervised learning without reconstruction decoders, exponential-moving-average teachers, or contrastive negatives. However, its isotropic task prior does not account for the cross-modal structure of multimodal tasks that read shared content. We introduce MM-LeJEPA by extending LeJEPA's task assumptions to multimodal learning. Specifically, we model multimodal tasks as strength-weighted combinations of components that read independent information from the two modalities and components that read the same content from both. Under explicit gradient-moment, correspondence, and symmetry assumptions, we derive a correlated Gaussian target that minimizes a weighted Fisher information functional under a fixed total variance constraint. An analytic transform adapts Sketched Isotropic Gaussian Regularization (SIGReg) to this target, yielding a two-term objective that combines distributional regularization with local-to-global prediction. The extension adds a single scalar hyperparameter, the target cross-modal correlation, and retains the simplicity of LeJEPA. On public audio-visual classification benchmarks, our results are competitive with representative open-source methods. Code will be released.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-17 · Indexed by Pub-AI: 2026-10-20 UTC

preprint

MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures

Haowen Bai, Rui Gong$, Meichen Liu, Ming-Xuan Weng, Bihan Wen, Kim-Hui Yap

Preprint posted: 2026-10-17 · Original record ↗

arxiv
2610.24681
Explicit source versions (1)
  • v1 · 2026-10-17
  1. Preprint available

    Source date: 2026-10-17 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

LeJEPA derives an isotropic Gaussian embedding target by minimizing, under its task assumptions, a Fisher information criterion that controls the task-averaged squared prediction bias. This enables unimodal self-supervised learning without reconstruction decoders, exponential-moving-average teachers, or contrastive negatives. However, its isotropic task prior does not account for the cross-modal structure of multimodal tasks that read shared content. We introduce MM-LeJEPA by extending LeJEPA's task assumptions to multimodal learning. Specifically, we model multimodal tasks as strength-weighted combinations of components that read independent information from the two modalities and components that read the same content from both. Under explicit gradient-moment, correspondence, and symmetry assumptions, we derive a correlated Gaussian target that minimizes a weighted Fisher information functional under a fixed total variance constraint. An analytic transform adapts Sketched Isotropic Gaussian Regularization (SIGReg) to this target, yielding a two-term objective that combines distributional regularization with local-to-global prediction. The extension adds a single scalar hyperparameter, the target cross-modal correlation, and retains the simplicity of LeJEPA. On public audio-visual classification benchmarks, our results are competitive with representative open-source methods. Code will be released.