Robotics · 2026-10-15

AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning

Jiawei Zhang, Chang Liu, Litao Deng, Tao Zhang, Qin Jin

arXiv:2610.20336 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AutoRSR converts scanned real workspaces into metrically aligned, robot-executable simulation assets and scalable manipulation data. Using multi-view observations, robot states, and open-vocabulary semantic queries, it reconstructs the scene, extracts object assets, registers to the robot base frame at metric scale, retargets manipulation from a demonstration, and augments via interpolation and domain randomization. Across three tasks, policies trained on 1K AutoRSR episodes achieve 73.3–86.7% real-world success.

Why it matters

The authors address a bottleneck in real2sim2real data generation: reconstruction alone still requires manual work for object asset preparation, scene-to-robot calibration, and executable trajectory creation. AutoRSR aims to reduce human effort by automating object extraction, metric alignment, and trajectory generation/augmentation from scanning inputs and semantic queries, producing reusable simulation assets for policy learning.

Method

Hybrid workspace scan while recording wrist-camera poses and robot joint states; use open-vocabulary semantic queries to reconstruct the workspace and extract individual object assets without manually annotated masks.
Automatically register the reconstructed scene to the robot base frame at metric scale using synchronized robot-camera geometry; retarget the demonstration video into executable robot manipulation trajectories.
Scale data via trajectory interpolation and domain randomization to generate diverse simulation episodes without additional real-robot interaction.

Limitation

The authors state that collision meshes provide geometric feasibility screening and do not guarantee force closure or dynamic task success.

Abstract (from arXiv)

Scaling embodied robot learning requires large and diverse manipulation datasets, but collecting real-world demonstrations remains expensive in human effort and robot time. Simulation provides a scalable alternative, yet converting a real workspace into a robot-learning-ready environment still requires substantial manual effort for object extraction, asset processing, coordinate calibration, and trajectory specification. We present AutoRSR, a highly automated real-to-sim-to real framework that converts scanned real-world workspaces into reusable embodied simulation assets and scalable manipulation data. Given multi-view observations, robot states, and open-vocabulary semantic queries, AutoRSR reconstructs the workspace, extracts individual object assets, and automatically registers the reconstructed scene to the robot base frame at metric scale. Starting from a small number of reference motions, AutoRSR then generates executable manipulation trajectories and scales them through trajectory interpolation and domain randomization. After workspace scanning and semantic queries, the remaining pipeline requires no manual object annotation, scene-to-robot calibration, or robot teleoperation. Across three real-world manipulation tasks, policies trained on 1K automatically generated episodes per task achieve 73.3--86.7% real-world success rates, comparable to pi 0.5 trained from 150 human teleoperated demonstrations, while reducing the human time required for data generation by an order of magnitude. These results demonstrate that real-world observations can be converted into metrically aligned, executable, and scalable simulation data with substantially reduced human effort.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC

preprint