Pub-AI: AI in Science

Driving · 2026-09-30

Real2Gym: Building Gyms from Videos, Bringing Skills to Robots

Kerui Ren, Yingxiang Xu, Kaiwen Song, Linning Xu, Bo Dai, Mulin Yu, Tao Lu

arXiv:2609.37089PDF

Auto-summarized: this summary was generated by a language model from the paper’s text and has not been reviewed by an editor. Check the paper before relying on it. Benchmark numbers appear only after a human has verified them.

TL;DR

Real2Gym is an agentic Real2Sim2Real framework that builds editable, physics-validated simulation gyms from human and robot demonstration videos. It reconstructs aligned Blender+MuJoCo scenes, validates/retargets actions via native physics, augments feasible task variations, and distills successful executions into reusable, object-relative skills for simulation and real robots without model weight updates.

Why it matters

The authors address the difficulty of turning real demonstration videos into executable robot skill libraries by combining (1) visually aligned scene reconstruction with (2) native-physics validation of demonstrated/retargeted actions, then (3) action-code execution plus feedback-driven skill extraction. They report improvements in simulation environment reconstruction and agent performance efficiency, and improved physical execution on four real Franka tasks.

Method

  • Real2Sim scene construction: reconstruct an editable 3D scene aligned to the demonstration using camera/scale constraints, segmentation, and iterative reprojection checks; import the target robot via URDF/MJCF and refine object/camera/robot alignment.
  • Action reconstruction & physical validation: recover interaction events, visually correct discrepancies across diagnostic dimensions, instantiate in MuJoCo, progressively calibrate execution, and validate by end-to-end native physics completion; reimport trajectories into Blender for frame-matched comparison.
  • Skill accumulation agent: in reconstructed gyms, generate executable Python code for manipulation subtasks, observe outcomes, and distill successful/failed execution into reusable skills (task procedures, object-relative motions, recovery guidance) while keeping underlying model weights fixed.
Abstract (from arXiv)

Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.

Related papers