Driving · 2026-10-15
Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution
Yunze Liu, Jing Wang, Shenrui Wu, Wenkun He, Guanqi Zhan, Xianzheng Ma, Junyu Chen, Wenhang Ge, Haozhan Tang, Yuchao Gu, Zhuoyang Zhang, Zekai Wang, Dantong Niu, Zhen Dong, Yuke Zhu, Trevor Darrell, Linxi "Jim" Fan, Song Han, Han Cai
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Robota-Embed is a vision–language–action (VLA) embedding model that jointly encodes current-state queries and recorded future continuations in a predictive retrieval setting. The authors train it on 1,695,684 filtered current–future pairs and evaluate on Robota-Embed-Bench (4,000 queries, 8,000-clip gallery), reporting higher retrieval accuracy with native action than action-free baselines. The same retrieval space is used for robot in-context learning (Embed2ICL) and task-local context evolution (Embed2Evolve).
Why it matters
The authors argue that retrieval for robot control must match task intent, execution stage, and what should follow. By embedding vision, language, and native action at both query and candidate endpoints, Robota-Embed targets this “predictive physical-experience retrieval” objective and then tests whether such retrieval improves downstream robot control tasks in both frozen- and evolving-policy settings.
Method
- Train Robota-Embed for current-to-future retrieval using predictive alignment (bidirectional InfoNCE), multimodal grounding and discrimination, and representation regularization; actions are mapped into the backbone with source-specific projectors.
- Build a multi-source training corpus (1,695,684 filtered current–future pairs) and evaluate on Robota-Embed-Bench (4,000 queries; shared 8,000-clip gallery) with clip-level and episode-level retrieval metrics.
- Use the shared retrieval space for two applications: Embed2ICL (training-free DreamZero warm-up and training-based Fast-WAM conditioning) and Embed2Evolve (evolving task-local memory/context around a frozen GPT-6 Astra policy).
Abstract (from arXiv)
Retrieving useful context for a robot requires matching its task, execution stage, and the continuation that should follow. We introduce Robota-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. Predictive retrieval training aligns these endpoints, enabling historical continuations to serve as context at inference time. We construct a multi-source corpus of 1,695,684 cleaned training anchors and introduce Robota-Embed-Bench, with 4,000 queries and a shared 8,000-entry gallery. Robota-Embed achieves 47.33% clip-level and 81.77% episode-level R@1, exceeding action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. Controlled retrieval studies show that removing native action degrades retrieval more than removing vision. We study two complementary applications. Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset, while improving task progress on real robots. Embed2Evolve uses execution feedback to grow task-local memory and refine context use around a frozen GPT-6 Astra policy. Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout, counting both successes and failures. These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution.
Related papers
- Fast-VLA: You Only Think Once for Robot Manipulation
- Causal Analysis of Robot Foundation Models Enables Efficiency Gains
- EvoMem-VLA: State-Evolution Memory for Long-Horizon Robot Manipulation
- Juno: Taming Predictive Latents for Vision-Language-Action Models
- Vision-Predictor-Action: Towards Efficient Embodied AI via Neuro-Symbolic JEPA Predictors
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.19495
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Retrieving useful context for a robot requires matching its task, execution stage, and the continuation that should follow. We introduce Robota-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. Predictive retrieval training aligns these endpoints, enabling historical continuations to serve as context at inference time. We construct a multi-source corpus of 1,695,684 cleaned training anchors and introduce Robota-Embed-Bench, with 4,000 queries and a shared 8,000-entry gallery. Robota-Embed achieves 47.33% clip-level and 81.77% episode-level R@1, exceeding action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. Controlled retrieval studies show that removing native action degrades retrieval more than removing vision. We study two complementary applications. Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset, while improving task progress on real robots. Embed2Evolve uses execution feedback to grow task-local memory and refine context use around a frozen GPT-6 Astra policy. Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout, counting both successes and failures. These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution.