Driving · 2026-11-10
World In-Context Learning for Agentic Robot Manipulation
Heng Fang, Zehui Guo, Yuehan Zhang, Mingyang Du, Dingkang Liang, Xiang Bai
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose World In-Context Learning (WICL): a frozen foundation model starts from a demonstration, rehearses in a world (e.g., simulator) to get feedback, and writes the feedback into its context rather than updating weights. They implement this in RoboScript, learning from a single demonstration per task via rehearsal, freezing a Task Memory, and deploying with data isolation.
Why it matters
The paper addresses a gap where in-context learning for robot manipulation has not matched trained policies because demonstrations from another scene lack world feedback. By rehearsal with world feedback and freezing the resulting semantic context, the method aims to generalize across visual distributions and transfer across settings and embodiments without gradient updates.
Method
- Define WICL: keep model weights frozen, initialize from a demonstration, rehearse in a world to obtain feedback, and update only the context (readable text).
- Implement RoboScript: from one expert demonstration, learn a Task Memory by imitation (S1) and rehearsal (S2), then freeze and deploy on unseen instances with no world feedback during evaluation.
- Enforce data isolation: acting in simulation and deployment uses only deployable robot inputs (camera images, depth, proprioception); simulator privileged information is used only as supervision and blocked by a memory linter.
Limitation
The method requires that the world be faithful to the task, resettable, and keep rehearsal instances apart from evaluation instances.
Abstract (from arXiv)
Trained action policies write manipulation skill into network weights, tying what is learned to the distribution it is learned from. In-context learning (ICL) avoids this coupling but has not matched trained policies, because a demonstration recorded in another scene carries no feedback from the world the robot currently faces. We observe that what an agent learns in context is semantic, recording task steps and object roles rather than visual appearance, so experience gathered in one world transfers to another. Based on this insight, we propose World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback into its context rather than its weights. We build RoboScript, a modular infrastructure that implements WICL for agentic manipulation. From a single demonstration, RoboScript learns each task through rehearsal in simulation, freezes the learned knowledge as a Task Memory, and deploys the frozen memory on new scenes with data isolation that ensures every deployment input is available on a real robot. On all 50 tasks of RoboTwin 2.0, RoboScript reaches 83.4% mean success across clean and randomized settings with one demonstration per task and no gradient updates, exceeding the strongest trained action policy by over 20 points. The method generalizes across visual distributions, improves through reflection, scales with the foundation model, and transfers to a physical dual-arm robot through cross-embodiment memory adaptation. With a world in the loop, a robot can learn a manipulation task in its context as well as it learns one in its weights. All code is available at https://github.com/ArcheOne-AI/RoboScript
Related papers
- In-Context Learning for Robots: Methods and Applications
- Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation
- Agentic World Action Models
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- SkillWeaver: Agentic Exploration over Neural Interaction Skills for Scalable Robot Data Generation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-09 · Indexed by Pub-AI: 2026-11-10 UTC
World In-Context Learning for Agentic Robot Manipulation
Preprint posted: 2026-11-09 · Original record ↗
- arxiv
- 2611.12885
Explicit source versions (1)
- v1 · 2026-11-09
- Preprint available
Source date: 2026-11-09 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.12885v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Trained action policies write manipulation skill into network weights, tying what is learned to the distribution it is learned from. In-context learning (ICL) avoids this coupling but has not matched trained policies, because a demonstration recorded in another scene carries no feedback from the world the robot currently faces. We observe that what an agent learns in context is semantic, recording task steps and object roles rather than visual appearance, so experience gathered in one world transfers to another. Based on this insight, we propose World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback into its context rather than its weights. We build RoboScript, a modular infrastructure that implements WICL for agentic manipulation. From a single demonstration, RoboScript learns each task through rehearsal in simulation, freezes the learned knowledge as a Task Memory, and deploys the frozen memory on new scenes with data isolation that ensures every deployment input is available on a real robot. On all 50 tasks of RoboTwin 2.0, RoboScript reaches 83.4% mean success across clean and randomized settings with one demonstration per task and no gradient updates, exceeding the strongest trained action policy by over 20 points. The method generalizes across visual distributions, improves through reflection, scales with the foundation model, and transfers to a physical dual-arm robot through cross-embodiment memory adaptation. With a world in the loop, a robot can learn a manipulation task in its context as well as it learns one in its weights. All code is available at https://github.com/ArcheOne-AI/RoboScript
