Video World Models · 2026-10-07
MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
Souptik Sen, Zahra Ahmadi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
MacJEPA is a missing-modality-robust audio-visual recognition model for untrimmed egocentric videos with temporally localized sensor outages. It uses interval-conditioned queries over multimodal untrimmed context, trains with window-local modality dropout using typed, timed masks, and repurposes JEPA’s masked-to-clean latent alignment as a supervised robustness objective jointly with recognition—without test-time adaptation.
Why it matters
The authors argue that real sensors fail and recover during long, untrimmed observations, creating local missingness that clip-level binary dropout cannot represent. MacJEPA targets this by modeling which modality is missing when, and training the model to keep interval-conditioned predictions consistent under simulated temporal outages across overlapping events.
Method
- Formulate interval-conditioned action/event recognition directly on the untrimmed audio-visual timeline; annotations are supplied via interval queries rather than inferred.
- Use window-local modality dropout: partition each observation into windows and substitute a learned absence embedding for the missing modality in that window (typed, timed masks).
- Train with content and query JEPA: align masked online latent representations to clean EMA teacher targets in a shared latent space, optimized jointly with classification; no test-time adaptation.
Abstract (from arXiv)
Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce \textbf{MacJEPA}, a missing-modality-robust \textbf{Ma}sked-\textbf{c}ontext query \textbf{JEPA} that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.
Related papers
- SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models
- EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
- MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures
- V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos
- EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model -- From Semantic Understanding through Visual Foresight to Action
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC
MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
Preprint posted: 2026-10-06 · Original record ↗
- arxiv
- 2610.08192
Explicit source versions (1)
- v2 · 2026-10-09
- Explicit source version detected · v2
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce \textbf{MacJEPA}, a missing-modality-robust \textbf{Ma}sked-\textbf{c}ontext query \textbf{JEPA} that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.
