Video World Models · 2026-10-07

MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos

Souptik Sen, Zahra Ahmadi

arXiv:2610.08192PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

MacJEPA is a missing-modality-robust audio-visual recognition model for untrimmed egocentric videos with temporally localized sensor outages. It uses interval-conditioned queries over multimodal untrimmed context, trains with window-local modality dropout using typed, timed masks, and repurposes JEPA’s masked-to-clean latent alignment as a supervised robustness objective jointly with recognition—without test-time adaptation.

Why it matters

The authors argue that real sensors fail and recover during long, untrimmed observations, creating local missingness that clip-level binary dropout cannot represent. MacJEPA targets this by modeling which modality is missing when, and training the model to keep interval-conditioned predictions consistent under simulated temporal outages across overlapping events.

Method

  • Formulate interval-conditioned action/event recognition directly on the untrimmed audio-visual timeline; annotations are supplied via interval queries rather than inferred.
  • Use window-local modality dropout: partition each observation into windows and substitute a learned absence embedding for the missing modality in that window (typed, timed masks).
  • Train with content and query JEPA: align masked online latent representations to clean EMA teacher targets in a shared latent space, optimized jointly with classification; no test-time adaptation.
Abstract (from arXiv)

Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce \textbf{MacJEPA}, a missing-modality-robust \textbf{Ma}sked-\textbf{c}ontext query \textbf{JEPA} that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC

preprint

MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos

Souptik Sen, Zahra Ahmadi

Preprint posted: 2026-10-06 · Original record ↗

arxiv
2610.08192
Explicit source versions (1)
  • v2 · 2026-10-09
  1. Explicit source version detected · v2

    Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
  2. Preprint available

    Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce \textbf{MacJEPA}, a missing-modality-robust \textbf{Ma}sked-\textbf{c}ontext query \textbf{JEPA} that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.