Unassigned · 2026-11-13

Spatial Foundry: A Multimodal Dataset for Spatial Intelligence

Ethan Lin, Yang Song

arXiv:2611.17318PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Spatial Foundry is a dataset of 5,280 furnished residential 3D scene configurations with aligned floor plans, perspective/panoramic RGB, metric depth, point clouds, meshes, instance-level 3D boxes, and camera trajectory–video pairs. Shared scene coordinates and editable source scenes support independent variation of camera motion vs scene edits for spatial perception and shared world modeling.

Why it matters

The authors report a way to connect visual observations to spatial geometry, object persistence, camera motion, and controlled scene changes under a common reference. By recording calibrated observations, persistent instance identities, trajectories with poses/timestamps, and explicit before/after edit records, the dataset enables reproducible comparisons and evaluation for tasks like floor-plan-conditioned 3D generation, reconstruction, instance perception, camera-conditioned video generation, and state-conditioned scene editing.

Method

  • Provide 5,280 furnished residential 3D configurations from U.S. priors: GPT-6 generates structured floor plans; human review checks spatial usability; Blender assembly with Infinigen assets produces editable source scenes.
  • Render aligned multimodal observations: perspective/panoramic RGB, calibrated metric depth, point clouds/meshes, instance-level 3D bounding boxes, and trajectory–video pairs with per-frame camera pose and timestamps.
  • Use a controlled organization principle: separate camera variation from environment variation, enabling fixed-scene camera sequences and fixed-camera before/after edit pairs; shared instance identifiers link cross-view and temporal correspondences.
Abstract (from arXiv)

Spatial intelligence requires connecting visual observations to the geometry, objects, and persistent state of an environment. Images alone leave these relationships implicit, while disconnected annotations make it difficult to study appearance, camera motion, and scene changes under a common reference. We introduce Spatial Foundry, a collection of 5,280 furnished residential 3D scene configurations with editable assets and aligned visual, geometric, and temporal data. Its modalities include floor plans, perspective and panoramic RGB, metric depth, point clouds, meshes, instance-level 3D bounding boxes, and camera trajectories paired with video. Shared scene coordinates link observations across views and time; editable source scenes enable additional camera paths and controlled object changes. We describe the data representation and construction workflow, and discuss applications in floor-plan-conditioned 3D generation, geometric reconstruction, instance-level perception, camera-conditioned video generation, shared world modeling, and scene editing. Controlled camera and scene variations support spatial perception and world-model research. The public project page provides videos and an interactive colored point-cloud preview.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-12 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Spatial Foundry: A Multimodal Dataset for Spatial Intelligence

Ethan Lin, Yang Song

Preprint posted: 2026-11-12 · Original record ↗

arxiv
2611.17318
Explicit source versions (1)
  • v1 · 2026-11-12
  1. Preprint available

    Source date: 2026-11-12 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Spatial intelligence requires connecting visual observations to the geometry, objects, and persistent state of an environment. Images alone leave these relationships implicit, while disconnected annotations make it difficult to study appearance, camera motion, and scene changes under a common reference. We introduce Spatial Foundry, a collection of 5,280 furnished residential 3D scene configurations with editable assets and aligned visual, geometric, and temporal data. Its modalities include floor plans, perspective and panoramic RGB, metric depth, point clouds, meshes, instance-level 3D bounding boxes, and camera trajectories paired with video. Shared scene coordinates link observations across views and time; editable source scenes enable additional camera paths and controlled object changes. We describe the data representation and construction workflow, and discuss applications in floor-plan-conditioned 3D generation, geometric reconstruction, instance-level perception, camera-conditioned video generation, shared world modeling, and scene editing. Controlled camera and scene variations support spatial perception and world-model research. The public project page provides videos and an interactive colored point-cloud preview.