Video World Models · 2026-11-06

PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation

Chuanxin Song, Peixi Peng

arXiv:2611.07873PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PVDepth is an end-to-end panoramic video depth estimation framework adapted from a perspective video depth model. It addresses ERP-specific geometric distortions and non-linear temporal dynamics using Progressive Sphere-aware Noise Initialization and Cube-rectified Temporal Modeling. The paper introduces PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, and reports improved geometric accuracy and temporal consistency. Code and data are released.

Why it matters

The authors focus on temporally consistent depth for panoramic videos under equirectangular projection (ERP), where both spatial sampling and temporal motion become distorted. By introducing geometry-aware noise initialization and cube-rectified temporal modeling, PVDepth targets these ERP-specific issues in an end-to-end way, and PanoCARLA alleviates the data scarcity bottleneck for panoramic RGB-D video depth estimation.

Method

  • Progressive Sphere-aware Noise Initialization: anneals the initial diffusion noise from planar to sphere-aware noise by using a variance-preserving mixture, to match ERP’s non-uniform information density.
  • Cube-rectified Temporal Modeling: adds an auxiliary cubemap temporal branch to model near-linear temporal dynamics in cube representation, then projects correction back to the ERP domain.
  • PVDepth builds on a pretrained video diffusion backbone (DepthCrafter) using VAE latents, frame-wise concatenation, and CLIP-guided cross-attention.

Limitation

The authors note that PVDepth struggles on strong reflections (e.g., reflective water): reflected structures are incorrectly interpreted as physical geometry, likely due to the absence of such patterns in the training data.

Abstract (from arXiv)

Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models. However, advancements in this field are impeded by two primary obstacles: the scarcity of large-scale training data and the unique spatiotemporal challenges of Equirectangular Projection (ERP), which hinder the direct transfer of perspective models. In this paper, we first present PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, featuring natural motion trajectories and drone-like roaming perspectives. Building on this foundation, we propose PVDepth, an end-to-end framework adapted from perspective video depth models. To tackle ERP-specific geometric distortions and consequent non-linear temporal dynamics, we introduce two core mechanisms: (1) A Progressive Sphere-aware Noise Initialization strategy that anneals the noise distribution from planar to spherical, guiding the model to adapt to non-uniform information density; and (2) A Cube-rectified Temporal Modeling module that incorporates an auxiliary cubemap temporal branch to rectify non-linear temporal dynamics in the ERP domain. Extensive experiments demonstrate that PVDepth achieves superior performance, generating geometrically accurate and temporally consistent depth sequences. Code and data will be released at https:// github.com/ChuanxinSong/PVDepth.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC

preprint

PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation

Chuanxin Song, Peixi Peng

First announced by arXiv: 2026-11-06 · Original record ↗

arxiv
2611.07873
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07873v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models. However, advancements in this field are impeded by two primary obstacles: the scarcity of large-scale training data and the unique spatiotemporal challenges of Equirectangular Projection (ERP), which hinder the direct transfer of perspective models. In this paper, we first present PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, featuring natural motion trajectories and drone-like roaming perspectives. Building on this foundation, we propose PVDepth, an end-to-end framework adapted from perspective video depth models. To tackle ERP-specific geometric distortions and consequent non-linear temporal dynamics, we introduce two core mechanisms: (1) A Progressive Sphere-aware Noise Initialization strategy that anneals the noise distribution from planar to spherical, guiding the model to adapt to non-uniform information density; and (2) A Cube-rectified Temporal Modeling module that incorporates an auxiliary cubemap temporal branch to rectify non-linear temporal dynamics in the ERP domain. Extensive experiments demonstrate that PVDepth achieves superior performance, generating geometrically accurate and temporally consistent depth sequences. Code and data will be released at https:// github.com/ChuanxinSong/PVDepth.