Driving · 2026-11-02

When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks

Alex Zhou, Junhao Wu, Xueyang Zhou, Yixiao Huo, Hongcheng Gao, Di Zhu, Zhenghao Zhang, Yongchao Chen

arXiv:2610.41857PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Simulation robot benchmarks can miscount simulator contact artifacts as policy failures. The authors trace “contact ejection” failures to PhysX 5.3.1 contact/distance threshold behavior and audit episodes with SimuGuard. With the constant fixed (PhysX 5.4.0), episodes with artifacts drop from 12 in 100 to none, and five container-task success rises from 89% to 98%.

Why it matters

Success rates in simulation thus count failures that the simulator causes. The authors argue benchmarks can score policy performance inaccurately when contact artifacts happen, including benchmark scripted demonstrations failing the same way. Their auditing plugin records physics steps, flags anomalous contact ejecting behavior, and replays/reruns episodes so users can verify whether failures stem from simulator contact handling. They also recommend reporting success with an artifact-conditional range when engines can’t be repaired.

Method

  • Introduce SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa, and RoboDojo that records every physics step, detects contact artifacts, replays episodes bit for bit, and reruns with one simulation parameter changed.
  • Define contact artifacts as motion starting at a contact that neither the robot’s contact nor gravity account for, detected via three stages using thresholds on motion, gravity explanation, and carrier explanations.
  • Use controlled reruns to trace the first anomalous contact to the physics engine’s collision/contact handling and measure how success rates change when the artifact cause is removed.
Abstract (from arXiv)

Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-30 · Indexed by Pub-AI: 2026-11-02 UTC

preprint

When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks

Alex Zhou, Junhao Wu, Xueyang Zhou, Yixiao Huo, Hongcheng Gao, Di Zhu, Zhenghao Zhang, Yongchao Chen

Preprint posted: 2026-10-30 · Original record ↗

arxiv
2610.41857
Explicit source versions (1)
  • v1 · 2026-10-30
  1. Preprint available

    Source date: 2026-10-30 · First detected: 2026-11-02 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.