JEPA · 2026-11-13

Asymmetric Multimodal Learning with Decomposed JEPAs

Yusuf Shihata

arXiv:2611.16639PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors show that cross-modal alignment as a sole objective can lose task-critical private and synergistic information under informational asymmetry, leading to semantic collapse on compositional VQA. They prove an “Alignment Redundancy Bound” limiting aligned latents to the redundant information atom. They introduce Decomposed JEPA (D-JEPA) with a shared core plus private fibers, averting collapse and improving downstream accuracy.

Why it matters

This paper formalizes when alignment-only training is inherently lossy and empirically connects it to compositional reasoning failures (e.g., CLEVR attributes/counting). The proposed D-JEPA architecture targets the missing unique/synergistic information by factorizing shared and private representations and auditing what each subspace contains, including causal fiber-swap interventions.

Method

  • Prove an Alignment Redundancy Bound: alignment-only Dirichlet energy minimization retains at most the redundant information atom Red(Z;X1,X2), annihilating unique modality information Unq(Z;Xm) and synergy Syn(Z;X1,X2).
  • Introduce D-JEPA: factorize latents into an aligned shared core and private modality-specific fibers; private fibers recover discarded information via contextual masked prediction and asymmetric cross-subspace independence.
  • Evaluate downstream reasoning and information/audit diagnostics (Probe-PID/Flow-PID) and causal fiber-swap interventions to localize recovered information in private fibers.

Limitation

The authors’ evaluation focuses on frozen ViT-S and BERT representations evaluated on CLEVR and synthetic manifolds at reduced scale, and D-JEPA narrows but does not close the gap to frozen-backbone baselines (e.g., joint D-JEPA accuracy is below raw text-only and raw joint).

Abstract (from arXiv)

Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a \emph{sole} objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an \textbf{Alignment Redundancy Bound}: alignment-only Dirichlet energy minimization retains at most the redundant information atom $\Red(Z; X_1, X_2)$, annihilating private modality-specific structure $\Unq(Z; X_m)$ and cross-modal synergy $\Syn(Z; X_1, X_2)$. We introduce \textbf{Decomposed JEPA (D-JEPA)}, which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Asymmetric Multimodal Learning with Decomposed JEPAs

Yusuf Shihata

First announced by arXiv: 2026-11-13 · Original record ↗

arxiv
2611.16639
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.16639v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a \emph{sole} objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an \textbf{Alignment Redundancy Bound}: alignment-only Dirichlet energy minimization retains at most the redundant information atom $\Red(Z; X_1, X_2)$, annihilating private modality-specific structure $\Unq(Z; X_m)$ and cross-modal synergy $\Syn(Z; X_1, X_2)$. We introduce \textbf{Decomposed JEPA (D-JEPA)}, which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.