Driving · 2026-10-06
Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation
Siyuan Liu, Miao Li, Haibao Yu, Haohong Lin, Qing Zhou, Bingbing Nie, Ding Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ControlPed generates photorealistic, motion-controllable vehicle–pedestrian safety-critical scenarios for end-to-end driving safety evaluation. It builds on HazardPed (from 10,352 traffic videos) to synthesize conflict trajectories, lift them into text-conditioned 3D human motions via diffusion, and render multi-view sensor observations using animatable 3D Gaussian Splatting avatars. Evaluating 88 rendered scenarios shows seven end-to-end driving models’ mean HDScore drops from 88.8 to 47.4.
Why it matters
The authors address a gap between trajectory-based generators (no raw visuals) and video-based approaches (limited controllability) for rare safety-critical interactions. ControlPed couples controllable trajectory and pedestrian action generation with photorealistic multi-view rendering, then uses the resulting scenarios to test end-to-end driving models and expose failure modes under dangerous pedestrian behaviors.
Method
- Build HazardPed from 10,352 traffic videos: extract vehicle–pedestrian conflict trajectories, align HD maps, and annotate 3D human motions (text-conditioned), yielding 422 scenarios and 857 hazardous motion clips.
- Generate hazardous pedestrian conflict trajectories with conditional diffusion guided by surrounding agents, map semantics, and goal heading; train in two stages and use a conflict controller to parameterize conflict timing/severity.
- Lift 2D trajectories into text- and action-conditioned full-body motions via motion diffusion with trajectory anchoring, then inject animatable pedestrians into OmniRe 3DGS scenes and render multi-view sensor videos for end-to-end evaluation.
Limitation
The authors note that ControlPed currently operates in an open-loop setup, and extending it to closed-loop interactive simulation is future work.
Abstract (from arXiv)
Evaluating end-to-end autonomous driving under rare, safety-critical vehicle-pedestrian interactions requires photorealistic, sensor-level scenarios. However, trajectory-based scenario generators cannot synthesize raw visual observations, whereas video-based approaches lack controllability. To bridge this gap, we present ControlPed, a novel framework that combines trajectory-level conflict synthesis with 3D Gaussian Splatting (3DGS) to generate photorealistic, motion-controllable safety-critical scenarios. Built upon HazardPed, a dataset derived from 10,352 traffic videos comprising 422 conflict trajectories, HD maps, and 857 annotated 3D human motions, ControlPed first generates conflict trajectories, lifts them into 3D human motion sequences via text-conditioned motion diffusion, and finally renders multi-view sensor observations using animatable 3DGS avatars. Safety evaluation in 88 rendered photorealistic scenarios reveals that seven leading end-to-end driving models suffer a severe performance drop, with their mean HDScore plunging from 88.8 to 47.4, exposing major failure modes under dangerous pedestrian behaviors. The dataset and testing benchmarks will be released to facilitate safety assessment of vehicle-pedestrian interactions.
Related papers
- SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation
- DriveWorld3D Evidence Constrained 3D World Modeling and Generative Simulation for Real-World Autonomous Driving
- DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
- GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation
- STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC
Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation
Preprint posted: 2026-10-05 · Original record ↗
- arxiv
- 2610.06171
Explicit source versions (1)
- v1 · 2026-10-05
- Preprint available
Source date: 2026-10-05 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Evaluating end-to-end autonomous driving under rare, safety-critical vehicle-pedestrian interactions requires photorealistic, sensor-level scenarios. However, trajectory-based scenario generators cannot synthesize raw visual observations, whereas video-based approaches lack controllability. To bridge this gap, we present ControlPed, a novel framework that combines trajectory-level conflict synthesis with 3D Gaussian Splatting (3DGS) to generate photorealistic, motion-controllable safety-critical scenarios. Built upon HazardPed, a dataset derived from 10,352 traffic videos comprising 422 conflict trajectories, HD maps, and 857 annotated 3D human motions, ControlPed first generates conflict trajectories, lifts them into 3D human motion sequences via text-conditioned motion diffusion, and finally renders multi-view sensor observations using animatable 3DGS avatars. Safety evaluation in 88 rendered photorealistic scenarios reveals that seven leading end-to-end driving models suffer a severe performance drop, with their mean HDScore plunging from 88.8 to 47.4, exposing major failure modes under dangerous pedestrian behaviors. The dataset and testing benchmarks will be released to facilitate safety assessment of vehicle-pedestrian interactions.