Robotics · 2026-10-06

SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models

Xiaodong Wang, Tianle Li, Chuanxin Song, Junliang Xie, Zhanmi Zhong, Suiying Wu, Peixi Peng

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SimForcing transfers simulation motion priors into real-domain action-conditioned robot world models using latent-motion distillation (matching temporal differences between adjacent video latents) and multi-block simulation conditioning with condition dropout. A simulation-conditioning classifier-free guidance scheme balances predictions from branches with and without simulation latents. The jointly trained model generates both simulation conditions and real-domain videos without an extra world model at inference.

Why it matters

The authors address how to use simulation for action-conditioned robot video prediction when appearance differences complicate sim-to-real transfer and imperfect simulation trajectories can mislead generation. SimForcing internalizes simulation-derived motion priors in latent space and controls reliance on simulation guidance, aiming for more reliable real-domain predictions and usefulness for downstream policy learning.

Method

Transfer motion knowledge via latent-motion distillation: align temporal changes between adjacent teacher and student video latents (minimize squared L2 distance between predicted latent motion sequences) while using a frozen simulation teacher.
Use multi-block simulation conditioning with condition dropout: inject simulation latents into every video transformer block through zero-initialized projections, sample a retention gate, and corrupt simulation latents during training.
At inference, apply simulation-conditioning classifier-free guidance: combine predictions from branches with and without simulation latent input to balance explicit simulation guidance and internalized motion priors.

Abstract (from arXiv)

Action-conditioned robot world models must respond precisely to robot trajectories while preserving realistic visual dynamics, yet learning both from heterogeneous robot videos remains challenging. Simulation offers structured motion supervision, but appearance differences hinder direct transfer, and inaccurate simulation predictions can misguide real-video generation. We present SimForcing, a simulation-guided framework that uses simulation both as a source of transferable motion knowledge and as a controllable reference for prediction. First, we transfer motion knowledge from a simulation teacher through latent-motion distillation, aligning temporal changes in latent space to internalize motion priors while mitigating the influence of appearance differences. Second, we introduce multi-block simulation conditioning with condition dropout to exploit predicted simulation trajectories without relying excessively on their accuracy. Our simulation-conditioning classifier-free guidance scheme unifies these two ideas by balancing predictions based on internalized motion knowledge with those additionally guided by simulation latents. The jointly trained student generates both simulation conditions and real-domain videos, requiring no additional world model at inference. On Bridge, SimForcing achieves the best PSNR, SSIM, LPIPS, and FVD among the compared methods without external embodied pretraining. Evaluation on InternData-A1 further supports its applicability across robot datasets. Moreover, using our trained world model to initialize a vision-language-action model improves LIBERO success, suggesting its utility for downstream policy learning. \url{https://github.com/Wang-Xiaodong1899/SimForcing}

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC

preprint