Video World Models · 2026-11-10

From Video to Digital Twin: An Exploratory Study of World Models and Gaussian Splatting on In-the-Wild Video

Somesh Mohapatra

arXiv:2611.10013PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study generative world models and locally optimized 3D Gaussian splats on an uncurated in-the-wild news walkthrough of a food bank warehouse (20 frames for a commercial world model). The world model recovered camera poses and metric depth for three areas and fused them into ~747k Gaussians. Held-out novel-view PSNR drops when training uses only 8 frames, and locally trained splats degrade quickly away from the filmed path. Generating unseen views failed on the provider side twice.

Why it matters

It provides an empirical account of what current world models and 3D Gaussian splatting recover from a single in-the-wild walkthrough video, including metric-scale recovery, cost/credit behavior, and held-out novel-view scores computed on unmasked pixels. It also reports that extracting a usable metric OpenUSD digital-twin asset requires corrections, and that provider-side failures prevent evaluation of extrapolation to unseen areas.

Method

  • Use a 3-step pipeline: extract blur-aware frames, query a commercial world model for Gaussian-splat worlds, convert outputs to metric, Y-up OpenUSD; optionally fill missing building geometry with a scripted facility model.
  • Reconstruct four zones locally with CPU-trained 3D Gaussian splatting (poses from structure-from-motion), using presenter masking; evaluate on held-out frames with PSNR/SSIM/LPIPS computed only in unmasked pixels.
  • Compare trainer variants (full vs fewer frames vs removing background fill and person-hull carving) to measure how few-view training and presenter-handling affect fidelity on held-out frames.

Limitation

The authors state the captured example requires hand-selected crops/placement and that residual error against the real building is not measured because no survey exists; additionally, generative view extrapolation could not be evaluated because requests failed on the provider’s side twice.

Abstract (from arXiv)

Generative world models and 3D Gaussian Splatting promise to turn ordinary video into explorable 3D scenes, but little is reported on how they behave on uncurated, in-the-wild footage. We study what these models recover when the only input is a publicly available news walkthrough of a food bank warehouse, with presenters walking through every shot and no drawings, scans or site visit, and we use the result as the geometry of a digital twin. A commercial world model given 20 frames recovered camera poses and metric depth for three areas, which we fused into about 747,000 Gaussian splats for 200 credits (20 US cents). Splat models optimized locally on a CPU, at no credit cost, were photorealistic near the filmed path but degraded quickly away from it. On held-out frames, scored in pixels not covered by people, they reached 21.2 and 23.1 dB PSNR from 64 and 43 training frames, and lost 3.8 and 5.4 dB when given only the 8 frames a world model accepts. Removing the trainer's background fill and person-hull carving raised held-out PSNR, showing that cleaning the space behind moving people costs fidelity in observed pixels while its gain lies where no ground truth exists. Asking the world model to generate views of areas the video never showed failed on the provider's side twice, so extrapolation could not be evaluated and unseen space was filled by a scripted model instead. We also describe the corrections needed to turn world-model output into a valid, metric Universal Scene Description (OpenUSD) asset with its native splat schema, which serves as the only interface to a data layer, a browser viewer, Blender and NVIDIA Omniverse. The findings suggest that today's models recover the observed parts of a real scene cheaply and at metric scale, while few-view fidelity and extrapolating unobserved space remain open. The work is an exploration of these methods, not a deployed product.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

From Video to Digital Twin: An Exploratory Study of World Models and Gaussian Splatting on In-the-Wild Video

Somesh Mohapatra

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.10013
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.10013v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Generative world models and 3D Gaussian Splatting promise to turn ordinary video into explorable 3D scenes, but little is reported on how they behave on uncurated, in-the-wild footage. We study what these models recover when the only input is a publicly available news walkthrough of a food bank warehouse, with presenters walking through every shot and no drawings, scans or site visit, and we use the result as the geometry of a digital twin. A commercial world model given 20 frames recovered camera poses and metric depth for three areas, which we fused into about 747,000 Gaussian splats for 200 credits (20 US cents). Splat models optimized locally on a CPU, at no credit cost, were photorealistic near the filmed path but degraded quickly away from it. On held-out frames, scored in pixels not covered by people, they reached 21.2 and 23.1 dB PSNR from 64 and 43 training frames, and lost 3.8 and 5.4 dB when given only the 8 frames a world model accepts. Removing the trainer's background fill and person-hull carving raised held-out PSNR, showing that cleaning the space behind moving people costs fidelity in observed pixels while its gain lies where no ground truth exists. Asking the world model to generate views of areas the video never showed failed on the provider's side twice, so extrapolation could not be evaluated and unseen space was filled by a scripted model instead. We also describe the corrections needed to turn world-model output into a valid, metric Universal Scene Description (OpenUSD) asset with its native splat schema, which serves as the only interface to a data layer, a browser viewer, Blender and NVIDIA Omniverse. The findings suggest that today's models recover the observed parts of a real scene cheaply and at metric scale, while few-view fidelity and extrapolating unobserved space remain open. The work is an exploration of these methods, not a deployed product.