Robotics · 2026-10-23
Learning Robot Behavior Priors from Non-Robotic Synthetic Data
Mateusz Odrowaz-Sypniewski, Michael Amir, Jasmine Bayrooti, Ajay Shankar, Amanda Prorok
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors test whether an in-context Transformer for robot action prediction can be pretrained solely on non-robotic, procedurally generated synthetic data from mathematical control abstractions (MDPs, contextual bandits, linear dynamical systems). They report that such a model achieves non-trivial zero-shot action prediction on unseen simulated and real robot datasets via in-context learning, and that specialized control priors improve over a generic causal tabular prior.
Why it matters
If useful robotic structure can be learned without embodied data, it could reduce the bottleneck of expensive real-robot collection for embodied foundation models. The paper compares specialized synthetic control priors against a generic tabular causal prior and against a compute-matched baseline trained on real robot data, showing different behavior on seen vs. unseen tasks and supporting the value of structured synthetic pretraining.
Method
- Formulate robot action prediction as in-context classification: given a context set of (state, action) pairs, predict held-out actions from a context set in a single forward pass with a Transformer.
- Pretrain on synthetic tasks sampled from three non-robotic generators: Markov Decision Processes (MDP), Contextual Bandits, and Linear Dynamical Systems (LDS), using randomly permuted feature dimensions and tokenizing actions into K=64 bins for robotic data.
- Compare against a generic tabular prior (random SCM/BNN/GP mechanisms) and against compute-matched embodied baselines trained on subsets of real robot data; evaluate with accuracy and NLL (and RCD for continuous-action settings).
Limitation
The authors caution that the severity of baseline collapse “is not necessarily a generalizable statement about training models on (cross-)embodied data—instead, it may well reflect the limitations of our current compute budget (see scale limitations, Section VI-B ).”
Abstract (from arXiv)
Real robot data, rich across tasks and morphologies, is often treated as the bottleneck for training robot learning models, including embodied foundation models that aim for zero-shot physical transfer. In this paper, we investigate the degree to which such models can acquire necessary structure using data from systems without embodiment. To this end, we test if inexpensive, procedurally generated synthetic data can yield useful priors before a model observes any real-world behavior. We study this through robot action prediction, a natural pretraining objective and a direct test of whether structure learned from arbitrary synthetic tasks transfers to embodied settings. We introduce three data generation pipelines that model mathematical abstractions of robotic control, and use them as pretraining priors for a Transformer. We demonstrate that a model trained purely on this non-robotic data attains non-trivial zero-shot prediction performance on unseen physical robot tasks via an in-context learning mechanism, and that our specialized control priors notably improve upon generic causal priors. We also compare against a compute-matched baseline trained on real robot data, and find that it outperforms our model but only on the datasets it was trained on. These findings suggest that procedurally generated, mathematical abstractions can yield useful robotic priors, offering a potential complement to expensive embodied data collection.
Related papers
- Causal world models under physical parameter shifts
- EmbodiedRSI: Active Continual Robot Learning Through Hypothesis-Guided Co-Evolution
- Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
- Copper-Policy: Focus on the Representation for Robust Robot Manipulation
- AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC
Learning Robot Behavior Priors from Non-Robotic Synthetic Data
Preprint posted: 2026-10-22 · Original record ↗
- arxiv
- 2610.31275
Explicit source versions (1)
- v1 · 2026-10-22
- Preprint available
Source date: 2026-10-22 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Real robot data, rich across tasks and morphologies, is often treated as the bottleneck for training robot learning models, including embodied foundation models that aim for zero-shot physical transfer. In this paper, we investigate the degree to which such models can acquire necessary structure using data from systems without embodiment. To this end, we test if inexpensive, procedurally generated synthetic data can yield useful priors before a model observes any real-world behavior. We study this through robot action prediction, a natural pretraining objective and a direct test of whether structure learned from arbitrary synthetic tasks transfers to embodied settings. We introduce three data generation pipelines that model mathematical abstractions of robotic control, and use them as pretraining priors for a Transformer. We demonstrate that a model trained purely on this non-robotic data attains non-trivial zero-shot prediction performance on unseen physical robot tasks via an in-context learning mechanism, and that our specialized control priors notably improve upon generic causal priors. We also compare against a compute-matched baseline trained on real robot data, and find that it outperforms our model but only on the datasets it was trained on. These findings suggest that procedurally generated, mathematical abstractions can yield useful robotic priors, offering a potential complement to expensive embodied data collection.