Robotics · 2026-10-02

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

Samuel Liu, Youngsun Kim, Martin Matak, Gilwoo Lee

arXiv:2610.00821PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors separate simulation grounding into two axes—world grounding (aligns simulator with the real system) and behavior grounding (aligns simulated trajectories with human motion)—and co-train on a dynamic dexterous pick-and-sort task. Fully grounded co-training raises success from 52% to 86%, and averaged across configurations world grounding improves success by 18 points while behavior grounding improves by 10. They report policies that mix real-derived and simulation-derived behavior within rollouts.

Why it matters

For real2sim2real co-training, it clarifies which type of sim mismatch most affects downstream policy performance, and when imitation of human motion helps. The authors also test post-trained foundation models and provide rollout/latent-space observations of when policies switch between real-like and sim-like behavior, informing how to prioritize simulator fidelity vs. human-motion alignment.

Method

  • Build a real2sim2real pipeline that independently sets world grounding and behavior grounding to grounded/ungrounded, creating a 2×2 factorial set of simulated datasets for co-training.
  • Train flow-matching transformer policies from 100 real teleoperation demonstrations plus about 1,500 simulated trajectories, and evaluate over 50 real-world trials including nominal and shifted conditions.
  • For mechanism analysis, run rollout/latent-space diagnostics and compare behavior switching under grounded vs. ungrounded settings; also test co-training on post-trained foundation models.

Limitation

The authors state the latent-space analysis is qualitative and rests on selected rollouts, including a diagnostic policy trained with far less real data than the main conditions. They also note that observations rarely fall in the simulated region in real-world rollouts, and that it is hard to attribute motions directly to real vs. simulated data; they present observations as illustrations rather than direct evidence. They add the study covers a single task with a limited number of trials per condition and many differences fall within overlapping confidence intervals; grounded world varies visual and dynamic fidelity together so individual contributions remain unseparated.

Abstract (from arXiv)

Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC

preprint

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

Samuel Liu, Youngsun Kim, Martin Matak, Gilwoo Lee

Preprint posted: 2026-09-30 · Original record ↗

arxiv
2610.00821
Explicit source versions (1)
  • v3 · 2026-10-12
  1. Preprint available

    Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
  2. Explicit source version detected · v3

    Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.