Pub-AI: AI in Science

Robotics · 2026-09-30

SkillWeaver: Agentic Exploration over Neural Interaction Skills for Scalable Robot Data Generation

He Zhu, Lusen Zhao, Kwan Man Cheng, Su Li, Katerina Fragkiadaki

arXiv:2609.36171PDF

Auto-summarized: this summary was generated by a language model from the paper’s text and has not been reviewed by an editor. Check the paper before relying on it. Benchmark numbers appear only after a human has verified them.

TL;DR

SkillWeaver is an agentic framework for scalable robot data generation. Given a task and a simulated environment, a VLM agent reasons about what to do next, invokes and parameterizes Neural Interaction Skills (NIS)—reusable closed-loop policies for contact-rich manipulation—and uses verifier-guided tree search with reflection and experience memory to discover and verify long-horizon behaviors. It scales to 39.1K demonstrations across 14.1K scenes and distills them into visuomotor policies for generalization and sim-to-real transfer.

Why it matters

The authors target the bottleneck of scaling robot demonstration collection by replacing teleoperation-heavy pipelines with simulation where behavior generation itself is treated as agentic search over reusable learned interaction tools. Their NIS abstraction aims to expose contact-rich, closed-loop manipulation skills to reasoning agents, while verifier-guided tree search and hierarchical verification provide intermediate feedback for long-horizon tasks. The generated verified traces are distilled into deployable visuomotor policies for transfer without additional teleoperation.

Method

  • Uses Neural Interaction Skills (NIS): reusable, parameterized, closed-loop policies for contact-rich manipulation (PICK, PLACE, OPEN, CLOSE), plus motion planning for collision-free free-space motion.
  • Employs a VLM agent that reasons over current state, selects and parameterizes NIS, executes them in simulation, and observes outcomes.
  • Organizes demonstration generation as verifier-guided tree search (Monte Carlo Tree Search) with a hierarchical verifier (sub-goal + progress) and semantic experience memory; verified trajectories become training data distilled into visuomotor policies.
Abstract (from arXiv)

Large-scale demonstrations have driven unprecedented progress in robot learning, yet collecting robot data through teleoperation is expensive and difficult to scale to diverse environments and long-horizon tasks. Simulation offers a scalable alternative, but existing data-generation pipelines often rely on open-loop controllers, scripted skill sequences, or task-specific programs. We introduce SkillWeaver, an agentic framework that autonomously generates robot experience by exploring over Neural Interaction Skills (NIS): reusable, parameterized, closed-loop policies that expose learned physical interaction capabilities to a reasoning agent. Given a task and a simulated environment, a VLM agent reasons about what to do next, invokes and parameterizes NIS to interact with the environment, observes their outcomes, and generates verification, reflection, and memory to guide subsequent exploration. We instantiate NIS as reinforcement-learned policies for closed-loop, contact-rich manipulation and organize exploration as verifier-guided tree search, enabling the agent to discover successful long-horizon behaviors without relying on predetermined execution pipelines. SkillWeaver scales autonomously to 39.1K demonstrations across 14.1K scenes, which we distill into visuomotor policies. Across simulation benchmarks and real-world manipulation, training on SkillWeaver-generated experience substantially improves generalization to novel objects, spatial configurations, tasks, and environments, and enables zero- and few-shot sim-to-sim and sim-to-real transfer. Our results suggest agentic exploration over neural interaction skills as a scalable alternative for robot data generation.

Related papers