Video World Models · 2026-10-22
OccJEPA: Occlusion as Masking for Amodal Video Segmentation
Genpei Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
OccJEPA treats occlusion as masking for amodal video segmentation. It drops tokens not covered by an object’s visible mask, uses a pre-trained V-JEPA 2.1 predictor to fill the masked latent features, and decodes them with a light head into amodal masks. The model is deterministic and reported as 33.5× faster than Diffusion-VAS, improving over it on SAIL-VOS and TAO-Amodal (including a held-out test split).
Why it matters
The authors target amodal video segmentation where hidden parts must be completed. Their approach avoids fine-tuned video diffusion, depth estimation, and sampling, instead repurposing a pre-trained self-supervised video world-model predictor as a completion module. The reported speed/accuracy gains and ablations attribute improvements to the predictor’s pre-trained weights rather than extra capacity, and it works with visible masks tracked by SAM2.
Method
- Treat occlusion as masking: keep latent tokens whose coverage by the visible mask satisfies ρ_i ≥ τ, drop others, and let a pre-trained V-JEPA 2.1 predictor fill the dropped positions.
- Decode observed+predicted token features into amodal masks with a light head (no occluder masks, no depth, no sampling), with a second parameter-free re-crop pass at test time.
- Train the predictor module from released weights, keep the encoder frozen except for LoRA adapters, and evaluate on SAIL-VOS and TAO-Amodal under Diffusion-VAS protocols.
Abstract (from arXiv)
Amodal video segmentation asks for the complete mask of an object in every frame, including the parts hidden behind other objects. The best current methods generate these masks with fine-tuned video diffusion models, which need many denoising steps and an extra depth network, and give a different answer for every sample. Self-supervised video world models of the JEPA family are trained on a closely related task: predicting the representation of masked space-time regions from the visible ones. We therefore treat occlusion as masking. The tokens that the object's visible mask does not cover are dropped, the pre-trained V-JEPA 2.1 predictor fills them in, and a light head decodes the observed and predicted features into amodal masks. The resulting model, OccJEPA (Occlusion-as-masking JEPA), is deterministic and 33.5x faster than the diffusion-based state of the art, and more accurate on SAIL-VOS, on TAO-Amodal, including a held-out test split (+4.00 AP50) and visible masks tracked by SAM2, and on Kubric. Controlled experiments trace the gain to the predictor's pre-trained weights rather than to its capacity, for world models from 87M to 1.85B parameters. Code is available at https://github.com/zgp110/occjepa.
Related papers
- JEPA Learns What the Mask Leaves Unrecoverable
- Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
- iSEE: Object Permanence Through Self-Supervision
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
- MEND: Label-Free Detection, Localisation, and Correction of Latent Hallucination in World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC
OccJEPA: Occlusion as Masking for Amodal Video Segmentation
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.29061
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Amodal video segmentation asks for the complete mask of an object in every frame, including the parts hidden behind other objects. The best current methods generate these masks with fine-tuned video diffusion models, which need many denoising steps and an extra depth network, and give a different answer for every sample. Self-supervised video world models of the JEPA family are trained on a closely related task: predicting the representation of masked space-time regions from the visible ones. We therefore treat occlusion as masking. The tokens that the object's visible mask does not cover are dropped, the pre-trained V-JEPA 2.1 predictor fills them in, and a light head decodes the observed and predicted features into amodal masks. The resulting model, OccJEPA (Occlusion-as-masking JEPA), is deterministic and 33.5x faster than the diffusion-based state of the art, and more accurate on SAIL-VOS, on TAO-Amodal, including a held-out test split (+4.00 AP50) and visible masks tracked by SAM2, and on Kubric. Controlled experiments trace the gain to the predictor's pre-trained weights rather than to its capacity, for world models from 87M to 1.85B parameters. Code is available at https://github.com/zgp110/occjepa.