Video World Models · 2026-10-19

ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis

Ge Ya Luo, Daniel Severo, Brian Karrer, Christopher Pal, Adriana Romero-Soriano, Michal Drozdzal

arXiv:2610.23732PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

ESC is a geometry-aware diffusion framework for monocular novel view synthesis under drastic camera changes. It jointly predicts target RGB and depth, conditioned on the source image/depth and target camera rays via Plücker ray conditioning. A Projection Consistency Constraint (PCC) enforces bidirectional 3D depth re-projection consistency, and inference uses PCC-based best-of-N selection to return the most geometrically aligned sample.

Why it matters

The authors address 2D diffusion ambiguity in novel view synthesis by grounding generation in explicit 3D geometry and using PCC to reduce geometric violations (e.g., missing vs hallucinated surfaces). They report better visual accuracy and 3D consistency than prior NVS baselines on DL3DV, RealEstate10K, and Tanks & Temples.

Method

  • ESC jointly generates target RGB and depth maps conditioned on a source image, rendered source depth, and Plücker ray maps for source/target camera geometry using a Qwen MM-DiT + Ray ControlNet, with depth tokens concatenated to visual tokens.
  • Projection Consistency Constraint (PCC) performs bidirectional depth reprojection between source and target using known poses; it masks invalid correspondences and uses a distance/log-space depth error where lower PCC indicates better multi-view geometric consistency.
  • At inference, ESC-PCC-BoN generates N independent samples with different noise seeds, evaluates each sample with PCC, and returns the sample minimizing geometric alignment error; for PCC-based selection the authors disable scale alignment.

Limitation

Because PCC is computed using inverse warp that is occlusion-naïve and bleeds across depth discontinuities, the authors state that the differentiable surrogate ℒ_PCC is not expected to match the hard z-buffer PCC metric numerically.

Abstract (from arXiv)

Generating 3D-consistent novel views from 2D diffusion models remains a fundamental challenge in computer vision. In this work, we introduce a framework that enhances geometric alignment for monocular view synthesis under drastic camera changes by jointly generating a target view and its corresponding depth map. By conditioning the diffusion pipeline on the single source image, source depth, and target camera rays, our approach forces the model to inherently reason in 3D space even under extreme viewpoint shifts. To guarantee spatial fidelity, we propose a novel per-pixel depth consistency constraint, which verifies and enforces 3D consistency across generated views. Our framework establishes a robust pipeline for geometrically consistent view synthesis, yielding superior visual accuracy and spatial alignment compared to current state-of-the-art methods. With this work, we pave the road towards robust 3D world models capable of understanding complex environments from a single view image.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-19 UTC

preprint

ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis

Ge Ya Luo, Daniel Severo, Brian Karrer, Christopher Pal, Adriana Romero-Soriano, Michal Drozdzal

Preprint posted: 2026-10-14 · Original record ↗

arxiv
2610.23732
Explicit source versions (1)
  • v1 · 2026-10-14
  1. Preprint available

    Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Generating 3D-consistent novel views from 2D diffusion models remains a fundamental challenge in computer vision. In this work, we introduce a framework that enhances geometric alignment for monocular view synthesis under drastic camera changes by jointly generating a target view and its corresponding depth map. By conditioning the diffusion pipeline on the single source image, source depth, and target camera rays, our approach forces the model to inherently reason in 3D space even under extreme viewpoint shifts. To guarantee spatial fidelity, we propose a novel per-pixel depth consistency constraint, which verifies and enforces 3D consistency across generated views. Our framework establishes a robust pipeline for geometrically consistent view synthesis, yielding superior visual accuracy and spatial alignment compared to current state-of-the-art methods. With this work, we pave the road towards robust 3D world models capable of understanding complex environments from a single view image.