Robotics · 2026-11-03

WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation

Jiaming Wang

arXiv:2611.00513PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

WorldSketch is an agentic robotics paradigm where a VLM builds a minimal kinematic world model (a “world sketch”) using perception tools and visual checks, then a geometric planner optimizes contact and whole-body motion on the sketch. Execution feedback verifies outcomes, updates uncertain joint hypotheses, and triggers sketch revisions. The authors report zero-shot success of 91% (RoboCasa), 97% (LIBERO-PRO), and 76% (R2R-CE), with fewer VLM input tokens.

Why it matters

The authors target a limitation of action-generation from VLMs (needing precise geometry/contact) by moving geometry/contact reasoning into an explicit, planner-actionable model. A sketch interface lets perception be task-directed and lets execution evidence correct the world model, rather than only producing actions or constraints directly from the VLM.

Method

  • A VLM constructs a minimal kinematic “world sketch” (task-relevant entities with semantics, measured geometry, joint kinematics, and joint hypotheses with priors) using perception-tool queries (e.g., points/views/label views) and visual depth checks.
  • A geometric planner then jointly optimizes contact and whole-body motion on the resulting sketch, including the gripper path along the relevant joint.
  • Execution feedback verifies outcomes (via depth and effect regions), updates joint hypotheses by Bayes’ rule, and either retries the next hypothesis or revises/reshadows the sketch in the same or a fresh VLM session.

Limitation

The authors note that failures for WorldSketch are often due to execution issues (e.g., slip, small or wide parts), with 89% of CaP failures being execution-related and 77% of CaP-Agent0 failures being execution-related; they also report that some failures happen when the sketch is right but the motion fails (root-cause breakdown).

Abstract (from arXiv)

Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation

Jiaming Wang

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.00513
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00513v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.