Robotics · 2026-10-02
Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training
Samuel Liu, Youngsun Kim, Martin Matak, Gilwoo Lee
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors separate simulation grounding into two axes—world grounding (aligns simulator with the real system) and behavior grounding (aligns simulated trajectories with human motion)—and co-train on a dynamic dexterous pick-and-sort task. Fully grounded co-training raises success from 52% to 86%, and averaged across configurations world grounding improves success by 18 points while behavior grounding improves by 10. They report policies that mix real-derived and simulation-derived behavior within rollouts.
Why it matters
For real2sim2real co-training, it clarifies which type of sim mismatch most affects downstream policy performance, and when imitation of human motion helps. The authors also test post-trained foundation models and provide rollout/latent-space observations of when policies switch between real-like and sim-like behavior, informing how to prioritize simulator fidelity vs. human-motion alignment.
Method
- Build a real2sim2real pipeline that independently sets world grounding and behavior grounding to grounded/ungrounded, creating a 2×2 factorial set of simulated datasets for co-training.
- Train flow-matching transformer policies from 100 real teleoperation demonstrations plus about 1,500 simulated trajectories, and evaluate over 50 real-world trials including nominal and shifted conditions.
- For mechanism analysis, run rollout/latent-space diagnostics and compare behavior switching under grounded vs. ungrounded settings; also test co-training on post-trained foundation models.
Limitation
The authors state the latent-space analysis is qualitative and rests on selected rollouts, including a diagnostic policy trained with far less real data than the main conditions. They also note that observations rarely fall in the simulated region in real-world rollouts, and that it is hard to attribute motions directly to real vs. simulated data; they present observations as illustrations rather than direct evidence. They add the study covers a single task with a limited number of trials per condition and many differences fall within overlapping confidence intervals; grounded world varies visual and dynamic fidelity together so individual contributions remain unseparated.
Abstract (from arXiv)
Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.
Related papers
- Rollback: Experience Backtracking in World Models for Policy Improvement
- Can World and Action Modeling Benefit Each Other under Unified Causal Training?
- pico-WAM: Evaluating Counterfactual Prediction and Planning Error in a Compact World Action Model
- Exploring Temporal Context-Aware Physical Grounding in Latent World Models
- Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00821
Explicit source versions (1)
- v3 · 2026-10-12
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v3
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.
