Video World Models · 2026-11-13
Multi-View 3D Reconstruction Using V-JEPA Features
Anwar Shamim, Jeffrin Sam
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study supervised RGB-only multi-view 3D reconstruction using a downstream architecture initialized from a V-JEPA 2.1 encoder. They compare global vs spatial camera estimation and a joint relative-objective configuration on CO3D (five-view windows) and an independently trained TUM RGB-D setting. Results show mixed gains: spatial cameras improve CO3D absolute trajectory error but hurt trajectory/depth relative to identity/global; relative objectives improve calibrated/aligned-shape metrics but worsen raw reconstruction.
Why it matters
The paper targets whether predictive video features can be adapted to explicit multi-view geometry, and emphasizes separating representation adaptation from aligned-shape and raw reconstruction effects. The authors report component/objective trade-offs (depth vs pose, aligned vs raw geometry) and discuss the need for isolated component and encoder comparisons before attributing reconstruction gains to predictive video pretraining.
Method
- Supervised RGB-only multi-view reconstruction using a V-JEPA 2.1 ViT-B/16 encoder, with explicit camera estimation, scene aggregation, epipolar transport, and residual pointmaps.
- Compare global vs spatial camera estimation; refinement is learned and uses bounds on tangent-space twist components, with first-view as the coordinate frame reference.
- Use a geometry-conditioned latent-view (G-LVP) distillation head and train with geometry, calibration, confidence, and auxiliary objectives; evaluation distinguishes raw vs similarity-aligned geometry.
Limitation
The authors state that their comparisons do not isolate component effects because architecture, supervision and model selection vary together; they also note the need for isolated component and encoder comparisons. They further note limited CO3D test coverage (only apple and cake) and that the experiments do not measure generalization over all CO3D categories.
Abstract (from arXiv)
Predictive video encoders provide dense visual representations, but adapting them to 3D reconstruction requires accurate depth, camera poses, and cross-view geometry. We study supervised RGB-only multi-view reconstruction using an architecture that combines video-pretrained features, camera estimation, scene aggregation, epipolar transport, and residual pointmaps. Experiments compare global and spatial camera regression with joint relative objectives on CO3D, alongside an independently trained TUM RGB-D model. On 168 CO3D five-view windows, the spatial-camera configuration achieves 29.3% lower absolute trajectory error than the global-camera configuration, but performs worse than identity poses in trajectory error and worse than the global configuration in depth error. The joint relative-objective configuration improves normalized intrinsics error and similarity-aligned Chamfer distance from 0.6447 to 0.5573, but increases raw Chamfer distance from 3.5535 to 4.4374. On 163 indoor windows, camera error is 0.3063 m versus 0.3694 m for identity poses, while depth error is 0.4279 versus 0.2516 for constant median depth. Refinement worsens indoor camera error, and additional experiments show competing depth and pose effects. Overall, the results distinguish representation adaptation, aligned shape, and raw reconstruction, highlighting the need for isolated component and encoder comparisons before attributing reconstruction gains to predictive video pretraining.
Related papers
- Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
- PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation
- SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space
- DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation
- SymRegFlow: Symmetry-Regularized Flow Matching for Video World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC
Multi-View 3D Reconstruction Using V-JEPA Features
First announced by arXiv: 2026-11-13 · Original record ↗
- arxiv
- 2611.15143
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.15143v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Predictive video encoders provide dense visual representations, but adapting them to 3D reconstruction requires accurate depth, camera poses, and cross-view geometry. We study supervised RGB-only multi-view reconstruction using an architecture that combines video-pretrained features, camera estimation, scene aggregation, epipolar transport, and residual pointmaps. Experiments compare global and spatial camera regression with joint relative objectives on CO3D, alongside an independently trained TUM RGB-D model. On 168 CO3D five-view windows, the spatial-camera configuration achieves 29.3% lower absolute trajectory error than the global-camera configuration, but performs worse than identity poses in trajectory error and worse than the global configuration in depth error. The joint relative-objective configuration improves normalized intrinsics error and similarity-aligned Chamfer distance from 0.6447 to 0.5573, but increases raw Chamfer distance from 3.5535 to 4.4374. On 163 indoor windows, camera error is 0.3063 m versus 0.3694 m for identity poses, while depth error is 0.4279 versus 0.2516 for constant median depth. Refinement worsens indoor camera error, and additional experiments show competing depth and pose effects. Overall, the results distinguish representation adaptation, aligned shape, and raw reconstruction, highlighting the need for isolated component and encoder comparisons before attributing reconstruction gains to predictive video pretraining.
