Driving · 2026-10-05
SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation
Yicheng Zhu, Linfeng Tian, Tianmu Zhao, Yang Chen, Fan Zuo, Tao Li, Zilin Bian
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SceneFactory-3D is a GPU-batched, physics-grounded multi-agent driving simulator that executes acceleration/steering via per-wheel suspension- and friction-limited forces on 3D heightfields with rigid contact. It enables matched physical counterfactuals by keeping scenario and controllers fixed while varying road conditions across parallel worlds. The authors evaluate controller sensitivity to 21 friction/grade conditions, finding reduced friction strongly degrades task outcomes and increases near-collisions.
Why it matters
The authors target a sim2real gap caused by prescribing kinematic/behavioral execution that ignores tire-road physics. SceneFactory-3D isolates how changes in road physics (spatial friction patches and 3D grade/terrain) affect closed-loop multi-vehicle behavior by running matched counterfactual worlds. This supports physically grounded safety evaluation of controllers under adverse conditions while keeping scenario and controller population fixed.
Method
- Per-wheel vehicle execution: suspension/tire forces are computed at each wheel-road contact using spatially varying friction and 3D heightfields with rigid contact, at each 240 Hz physics step.
- Matched physical counterfactuals: the same scenario, initial state, routes, and controller population are run in parallel worlds that differ only in declared road conditions (terrain heightfield and friction patches).
- Canonical-route interface: controllers receive a shared route reference (planar route with route-relative samples and traffic state) while execution diverges through the physics.
Limitation
The paper notes that vehicle grip factor is set (μ_vehicle = 1.05) and road conditions are not described as a weather model (Condition parameters describe surface grip and grade; they are not a weather model).
Abstract (from arXiv)
Scalable driving simulators typically execute vehicle commands using prescribed behavioral or kinematic rules, overlooking the physics of tire-road interfaces, thereby limiting their ability to capture how adverse road and environmental conditions alter vehicle execution and propagate through traffic. To address this limitation, we present SceneFactory-3D, a GPU-batched, physics-grounded multi-agent driving simulator. Vehicles execute acceleration and steering commands via suspension- and friction-limited forces evaluated at each wheel-contact point. Spatially varying friction, per-world 3D heightfields, gravity, and rigid contact consistently govern wheel motion and chassis collisions. Per-world terrain isolation and GPU batching enable SceneFactory-3D to run matched physical counterfactuals in parallel: traffic scenario setup and vehicle controllers remain fixed while only the road condition changes, enabling the resulting closed-loop effects to be evaluated across parallel worlds. To demonstrate the advantage of the SceneFactory-3D-enabled counterfactual evaluation, we conduct an empirical study on vehicle controllers' sensitivity to road conditions. We study three learned-policy families on 1,024 matched 12-vehicle worlds per condition, and two classical planners on a shared 32-world subset, across 21 friction and grade conditions. When friction drops from 1.0 to 0.18, the share of vehicles that clear the work zone safely falls by 6 to 90 percentage points across learned policies (18-19 for classical planners), and near-collision situations become more frequent for every learned policy. Code: https://github.com/SmallWorldLab/SceneFactory_3D
Related papers
- DriveWorld3D Evidence Constrained 3D World Modeling and Generative Simulation for Real-World Autonomous Driving
- VehicleArena: A Realistic Urban Environment for Multi-Agent Driving
- Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation
- DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
- PhysWAM: Physically Consistent World Action Model for Autonomous Driving
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.02874
Explicit source versions (1)
- v2 · 2026-10-13
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v2
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
Scalable driving simulators typically execute vehicle commands using prescribed behavioral or kinematic rules, overlooking the physics of tire-road interfaces, thereby limiting their ability to capture how adverse road and environmental conditions alter vehicle execution and propagate through traffic. To address this limitation, we present SceneFactory-3D, a GPU-batched, physics-grounded multi-agent driving simulator. Vehicles execute acceleration and steering commands via suspension- and friction-limited forces evaluated at each wheel-contact point. Spatially varying friction, per-world 3D heightfields, gravity, and rigid contact consistently govern wheel motion and chassis collisions. Per-world terrain isolation and GPU batching enable SceneFactory-3D to run matched physical counterfactuals in parallel: traffic scenario setup and vehicle controllers remain fixed while only the road condition changes, enabling the resulting closed-loop effects to be evaluated across parallel worlds. To demonstrate the advantage of the SceneFactory-3D-enabled counterfactual evaluation, we conduct an empirical study on vehicle controllers' sensitivity to road conditions. We study three learned-policy families on 1,024 matched 12-vehicle worlds per condition, and two classical planners on a shared 32-world subset, across 21 friction and grade conditions. When friction drops from 1.0 to 0.18, the share of vehicles that clear the work zone safely falls by 6 to 90 percentage points across learned policies (18-19 for classical planners), and near-collision situations become more frequent for every learned policy. Code: https://github.com/SmallWorldLab/SceneFactory_3D