Video World Models · 2026-10-01
VR-JEPA: Learning Contrastive-State Latent Guidance for Generation-based Video Reasoning
Zehua Ma, Kun Xiang, Yunshuang Nie, Quanlin Chen, Haoyuan Li, Xiuwei Chen, Jiang Ji, Haijun Wu, Zhenyu Xie, Michael Kampffmeyer, Hanhui Li, Xiaodan Liang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
VR-JEPA uses V-JEPA latent predictions to guide generation-based video reasoning. It trains a latent reasoner with localized contrastive-state learning by comparing successful reference trajectories to task-matched generated candidates, and uses a skill-routed mixture of experts (SR-MoE) to specialize across reasoning skills. The predicted latent trajectories condition a video diffusion generator to improve task consistency.
Why it matters
The authors argue that video generators can look realistic yet produce physically/structurally inconsistent trajectories that fail reasoning. VR-JEPA explicitly models task-relevant state evolution in V-JEPA latent space and uses those predicted latent trajectories to guide synthesis, aiming to improve logical consistency and reduce physical artifacts on visual reasoning benchmarks.
Method
- Predict task-relevant latent trajectories from frozen V-JEPA features using a latent reasoner, then condition a video generator on the rollout.
- Train with localized contrastive-state learning (LCL): pair successful trajectories with task-matched generated candidates and weight contrastive supervision using discrepancies in dense V-JEPA feature space at state and token levels.
- Use skill-routed mixture of experts (SR-MoE) with 7 experts (VIS, NUM, IDN, SPA, DYN, CMP, ABS) and a state-conditioned Entmax 1.5 router to adapt expert combinations across reasoning skills.
Limitation
The provided text does not state a specific limitation.
Abstract (from arXiv)
Reasoning through video generation offers a promising path toward visual intelligence by modeling latent visual states and their dynamics. However, current video generation models often lack explicit guidance on how these states should evolve, leaving generated trajectories prone to physical and structural inconsistencies that undermine reasoning reliability. While the Video Joint-Embedding Predictive Architecture (V-JEPA) provides rich spatiotemporal priors learned through latent prediction, these general priors do not naturally adapt to the logical reasoning capabilities required for complex visual tasks. To bridge this gap, we propose VR-JEPA, a framework that aligns the V-JEPA predictor with task-specific reasoning logic through localized contrastive-state learning and uses its predicted latent trajectories to guide video generation for visual reasoning. Specifically, (i) we pair successful trajectories with generated alternatives under the same input conditions and use discrepancies in their V-JEPA representations to identify informative states and tokens for localized contrastive supervision. (ii) We further equip the V-JEPA predictor with skill-specific experts trained on anchor-task data, allowing the model to adaptively specialize its shared spatiotemporal priors across diverse cognitive domains. Together with skill-specific experts, this contrastive supervision enables VR-JEPA to predict latent trajectories that provide task-specific logical guidance for video generation. Comprehensive experiments on the large-scale VBVR-Pro-Bench dataset demonstrate that VR-JEPA achieves an $11.33\%$ relative improvement over the cutting-edge generation-based reasoning baseline, significantly mitigating physical artifacts and enhancing logical consistency.
Related papers
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness
- Juno: Taming Predictive Latents for Vision-Language-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC
VR-JEPA: Learning Contrastive-State Latent Guidance for Generation-based Video Reasoning
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2609.40129
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reasoning through video generation offers a promising path toward visual intelligence by modeling latent visual states and their dynamics. However, current video generation models often lack explicit guidance on how these states should evolve, leaving generated trajectories prone to physical and structural inconsistencies that undermine reasoning reliability. While the Video Joint-Embedding Predictive Architecture (V-JEPA) provides rich spatiotemporal priors learned through latent prediction, these general priors do not naturally adapt to the logical reasoning capabilities required for complex visual tasks. To bridge this gap, we propose VR-JEPA, a framework that aligns the V-JEPA predictor with task-specific reasoning logic through localized contrastive-state learning and uses its predicted latent trajectories to guide video generation for visual reasoning. Specifically, (i) we pair successful trajectories with generated alternatives under the same input conditions and use discrepancies in their V-JEPA representations to identify informative states and tokens for localized contrastive supervision. (ii) We further equip the V-JEPA predictor with skill-specific experts trained on anchor-task data, allowing the model to adaptively specialize its shared spatiotemporal priors across diverse cognitive domains. Together with skill-specific experts, this contrastive supervision enables VR-JEPA to predict latent trajectories that provide task-specific logical guidance for video generation. Comprehensive experiments on the large-scale VBVR-Pro-Bench dataset demonstrate that VR-JEPA achieves an $11.33\%$ relative improvement over the cutting-edge generation-based reasoning baseline, significantly mitigating physical artifacts and enhancing logical consistency.