Driving · 2026-10-05

Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Yuxing Wang, Yizhou Wang, Anqi Li, Shuo Wang, Sameer Satish Pusegaonkar, Haoquan Liang, Jiajun Li, Shenxin Jiang, Jianhe Yuan, Shangru Li, Tongwei Dai, Zihao Chen, David C. Anastasiu, Sujit Biswas, Xunlei Wu, Zheng Tang

arXiv:2610.02580PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Physical AI Smart Spaces (PAISS) is a public benchmark for indoor multi-camera, multi-class 3D perception. It provides synchronized 1080p video (up to 282.5 hours) from 1,799 cameras with automatic multi-camera identities, 2D/3D boxes, camera calibration, and depth where available, spanning synthetic generation (Isaac Sim), Cosmos Transfer 2.5 augmentation, and a hidden real-world Sim2Real target. It introduces a 3D HOTA protocol for 3D tracking evaluation.

Why it matters

The authors report PAISS as a large-scale evaluation resource aiming to test whether multi-camera 3D perception generalizes from synthetic and appearance-augmented data to deployment-like conditions. It includes geometry-aware annotations (global world-frame 3D boxes, calibration, depth where provided) and a hidden real-world target evaluated with a 3D instantiation of Higher Order Tracking Accuracy (HOTA) that accounts for detection and association in 3D locations/boxes.

Method

  • Provides synchronized multi-camera RGB (and optional depth) with automatic multi-camera identity labels, 2D/3D bounding boxes, and per-camera calibration in a shared global world coordinate system.
  • Uses synthetic generation with NVIDIA Omniverse/Isaac Sim and appearance-augmented data via Cosmos Transfer 2.5 (CT2.5), plus a hidden real-world target with time synchronization, VGGT-based calibration, and a 3D labeling UI for cross-view verification.
  • Evaluates multi-camera 3D detection and tracking using 3D HOTA (3D locations for 2024; 3D IoU between boxes for 2025/2026) computed per class and aggregated across scenes via the public evaluation server/leaderboard protocol.

Limitation

The dataset is bounded in scope and intended for careful research use, with a stated focus on indoor smart spaces captured by static, pre-calibrated 1080p cameras dominated by warehouse environments. The authors also state the synthetic avatars are drawn from a finite Omniverse asset library and are not suitable for fairness/demographic studies, and that it is not authorized for biometric, facial recognition, or surveillance use against real individuals.

Abstract (from arXiv)

Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-05 UTC

preprint

Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Yuxing Wang, Yizhou Wang, Anqi Li, Shuo Wang, Sameer Satish Pusegaonkar, Haoquan Liang, Jiajun Li, Shenxin Jiang, Jianhe Yuan, Shangru Li, Tongwei Dai, Zihao Chen, David C. Anastasiu, Sujit Biswas, Xunlei Wu, Zheng Tang

Preprint posted: 2026-10-01 · Original record ↗

arxiv
2610.02580
Explicit source versions (1)
  • v1 · 2026-10-01
  1. Preprint available

    Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.