Driving · 2026-11-02
When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks
Alex Zhou, Junhao Wu, Xueyang Zhou, Yixiao Huo, Hongcheng Gao, Di Zhu, Zhenghao Zhang, Yongchao Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Simulation robot benchmarks can miscount simulator contact artifacts as policy failures. The authors trace “contact ejection” failures to PhysX 5.3.1 contact/distance threshold behavior and audit episodes with SimuGuard. With the constant fixed (PhysX 5.4.0), episodes with artifacts drop from 12 in 100 to none, and five container-task success rises from 89% to 98%.
Why it matters
Success rates in simulation thus count failures that the simulator causes. The authors argue benchmarks can score policy performance inaccurately when contact artifacts happen, including benchmark scripted demonstrations failing the same way. Their auditing plugin records physics steps, flags anomalous contact ejecting behavior, and replays/reruns episodes so users can verify whether failures stem from simulator contact handling. They also recommend reporting success with an artifact-conditional range when engines can’t be repaired.
Method
- Introduce SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa, and RoboDojo that records every physics step, detects contact artifacts, replays episodes bit for bit, and reruns with one simulation parameter changed.
- Define contact artifacts as motion starting at a contact that neither the robot’s contact nor gravity account for, detected via three stages using thresholds on motion, gravity explanation, and carrier explanations.
- Use controlled reruns to trace the first anomalous contact to the physics engine’s collision/contact handling and measure how success rates change when the artifact cause is removed.
Abstract (from arXiv)
Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.
Related papers
- World Models Dream of Success: Diagnosing and Repairing Failure Insensitivity in Robot World Models
- RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
- Don't Throw Away the Tail: Action Upcycling for Policy Acceleration
- ForkCast: Predictive Recovery for Vision-Language-Action Policies via Counterfactual Futures
- RoboCoach: World Models as Active Coaches for Compositional Robot Skills
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-30 · Indexed by Pub-AI: 2026-11-02 UTC
When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks
Preprint posted: 2026-10-30 · Original record ↗
- arxiv
- 2610.41857
Explicit source versions (1)
- v1 · 2026-10-30
- Preprint available
Source date: 2026-10-30 · First detected: 2026-11-02 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.41857v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.
