Video World Models · 2026-11-06
PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation
Chuanxin Song, Peixi Peng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
PVDepth is an end-to-end panoramic video depth estimation framework adapted from a perspective video depth model. It addresses ERP-specific geometric distortions and non-linear temporal dynamics using Progressive Sphere-aware Noise Initialization and Cube-rectified Temporal Modeling. The paper introduces PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, and reports improved geometric accuracy and temporal consistency. Code and data are released.
Why it matters
The authors focus on temporally consistent depth for panoramic videos under equirectangular projection (ERP), where both spatial sampling and temporal motion become distorted. By introducing geometry-aware noise initialization and cube-rectified temporal modeling, PVDepth targets these ERP-specific issues in an end-to-end way, and PanoCARLA alleviates the data scarcity bottleneck for panoramic RGB-D video depth estimation.
Method
- Progressive Sphere-aware Noise Initialization: anneals the initial diffusion noise from planar to sphere-aware noise by using a variance-preserving mixture, to match ERP’s non-uniform information density.
- Cube-rectified Temporal Modeling: adds an auxiliary cubemap temporal branch to model near-linear temporal dynamics in cube representation, then projects correction back to the ERP domain.
- PVDepth builds on a pretrained video diffusion backbone (DepthCrafter) using VAE latents, frame-wise concatenation, and CLIP-guided cross-attention.
Limitation
The authors note that PVDepth struggles on strong reflections (e.g., reflective water): reflected structures are incorrectly interpreted as physical geometry, likely due to the absence of such patterns in the training data.
Abstract (from arXiv)
Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models. However, advancements in this field are impeded by two primary obstacles: the scarcity of large-scale training data and the unique spatiotemporal challenges of Equirectangular Projection (ERP), which hinder the direct transfer of perspective models. In this paper, we first present PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, featuring natural motion trajectories and drone-like roaming perspectives. Building on this foundation, we propose PVDepth, an end-to-end framework adapted from perspective video depth models. To tackle ERP-specific geometric distortions and consequent non-linear temporal dynamics, we introduce two core mechanisms: (1) A Progressive Sphere-aware Noise Initialization strategy that anneals the noise distribution from planar to spherical, guiding the model to adapt to non-uniform information density; and (2) A Cube-rectified Temporal Modeling module that incorporates an auxiliary cubemap temporal branch to rectify non-linear temporal dynamics in the ERP domain. Extensive experiments demonstrate that PVDepth achieves superior performance, generating geometrically accurate and temporally consistent depth sequences. Code and data will be released at https:// github.com/ChuanxinSong/PVDepth.
Related papers
- SPW-Nav: A Streaming Panoramic World Model for Language-Guided Navigation
- GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation
- Multi-View 3D Reconstruction Using V-JEPA Features
- Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering
- TAP-Out: Tracking Any Point in 360{\deg} via Outpainting
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC
PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation
First announced by arXiv: 2026-11-06 · Original record ↗
- arxiv
- 2611.07873
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07873v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models. However, advancements in this field are impeded by two primary obstacles: the scarcity of large-scale training data and the unique spatiotemporal challenges of Equirectangular Projection (ERP), which hinder the direct transfer of perspective models. In this paper, we first present PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, featuring natural motion trajectories and drone-like roaming perspectives. Building on this foundation, we propose PVDepth, an end-to-end framework adapted from perspective video depth models. To tackle ERP-specific geometric distortions and consequent non-linear temporal dynamics, we introduce two core mechanisms: (1) A Progressive Sphere-aware Noise Initialization strategy that anneals the noise distribution from planar to spherical, guiding the model to adapt to non-uniform information density; and (2) A Cube-rectified Temporal Modeling module that incorporates an auxiliary cubemap temporal branch to rectify non-linear temporal dynamics in the ERP domain. Extensive experiments demonstrate that PVDepth achieves superior performance, generating geometrically accurate and temporally consistent depth sequences. Code and data will be released at https:// github.com/ChuanxinSong/PVDepth.
