Driving · 2026-10-08
RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang, Zijian Du, Yang Liu, Jiaolong Yang, Li Fei-Fei, Jiajun Wu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RoboRender converts simulation trajectories into photorealistic, robot-oriented multi-view RGB videos for zero-shot visual sim-to-real policy transfer. The model is conditioned on simulated depth, language instructions, and robot RGB masks, preserving geometry, robot motion, and simulator action labels while synthesizing realistic textures, backgrounds, and distractors. Policies trained on RoboRender videos achieve 71% average real-world success, outperforming raw simulation and visual domain randomization.
Why it matters
The paper targets the visual sim-to-real gap for vision-based robot manipulation, arguing that intermediate representations can be semantically lossy or require extra perception modules. By generating full photorealistic RGB-action trajectories directly from simulator rollouts while keeping grounded structure and known action labels, RoboRender aims to enable zero-shot sim-to-real transfer across multiple task types and embodiments.
Method
- Train a robot-oriented video generation model conditioned on per-view simulated depth, CLIP-encoded language prompts, and robot RGB masks; denoise a multi-view latent to synthesize photorealistic RGB videos while preserving simulator-provided structure and action labels.
- Generate simulation trajectories with recorded multi-view RGB-D observations, proprioception, actions, and robot segmentation masks; synthesize Dgen by rendering these trajectories into RGB videos via RoboRender, then train policies on paired simulator states/actions.
- Add controllable distractors by masking depth patches during training so the model can synthesize prompt-guided clutter consistently across views, improving robustness to clutter.
Limitation
The paper notes that visual domain randomization still fails on visually precise tasks (opening tasks) because coarse appearance randomization is insufficient to close the visual sim-to-real gap.
Abstract (from arXiv)
Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.
Related papers
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization
- SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models
- Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
- AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC
RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.09254
Explicit source versions (1)
- v1 · 2026-10-07
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.