Driving · 2026-11-10

Library of Hands: Fix the Hands, Not the World Model

Hasaan Ahmad, Qizhen Ying, Yuxuan Sun, Tianjin Huang, Guangming Wang

arXiv:2611.12678PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Library of Hands (LoH) keeps a frozen world model (e.g., LeWM) and learns only a training-free correction from planner commands to applied actions (“the hand”). It reads the hand by finding which action best explains the change between two camera images under the frozen model, stores each identified hand, and recalls returning hands. In closed-loop simulation, LoH more than doubles success on three tasks after returns (28.6% to 62.3%) while losing only 0.6 pp when the hand never changes (77.5% vs 78.1%).

Why it matters

The authors argue that when cameras move or controllers change, planning can fail even if the world model remains accurate, because the planner uses commands under the wrong action convention. LoH addresses this without retraining or test-time weight updates: it identifies and reuses a correction map (“hand”) via evidence from the frozen predictor and a library for returning set-ups. This targets robustness for repeated lab deployments with recurring set-ups, aiming to avoid the calibration/retraining cycles required by prior approaches.

Method

  • LoH keeps the world model frozen and “reads” the applied action by asking the frozen predictor which action best explains the change between two successive camera images (after a block), using a surprise threshold to accept reliable readings.
  • It fits a correction map (“hand”) from commands to applied actions with reliability-weighted uncertainty from the predictor’s Jacobian, then chooses the map by exact Bayesian evidence across candidate map classes and delays.
  • It stores every identified hand in a library; when a set-up returns, LoH recalls the stored hand rather than re-identifying it, and replans by passing candidate actions through the chosen map and delay.

Limitation

The authors note that hands with dead bands (small commands have no effect) or saturation (large commands are capped) do not fit their action-mapping assumption, and tests them outside Equation 1.

Abstract (from arXiv)

A world model predicts how a scene changes after an action, and a planner uses it to search for actions that reach a goal. If the camera moves or the controller changes, the same command moves the robot differently: when its commands are turned by 90 degrees, a frozen planner with the LeWM world model that succeeds in 77.5% of episodes succeeds in only 7.8%. Current fixes retrain the model, need training data, or forget a set-up when it returns. We propose Library of Hands (LoH), a training-free method that keeps the world model frozen and corrects only the map from commands to applied actions, which we call the hand. LoH reads the hand through the frozen model, with no extra sensor: it finds the action that best explains the change between two camera images (a reading). It stores every hand it identifies, so a returning set-up is recognised rather than re-learnt. In closed-loop tests, LoH more than doubles success on three simulated tasks, from 28.6% to 62.3%, and loses only 0.6 percentage points when nothing changes. It also works, unchanged, with two other world models and on two of three models pretrained on real robot data, and its gain grows with the quality of its readings.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-09 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

Library of Hands: Fix the Hands, Not the World Model

Hasaan Ahmad, Qizhen Ying, Yuxuan Sun, Tianjin Huang, Guangming Wang

Preprint posted: 2026-11-09 · Original record ↗

arxiv
2611.12678
Explicit source versions (1)
  • v1 · 2026-11-09
  1. Preprint available

    Source date: 2026-11-09 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

A world model predicts how a scene changes after an action, and a planner uses it to search for actions that reach a goal. If the camera moves or the controller changes, the same command moves the robot differently: when its commands are turned by 90 degrees, a frozen planner with the LeWM world model that succeeds in 77.5% of episodes succeeds in only 7.8%. Current fixes retrain the model, need training data, or forget a set-up when it returns. We propose Library of Hands (LoH), a training-free method that keeps the world model frozen and corrects only the map from commands to applied actions, which we call the hand. LoH reads the hand through the frozen model, with no extra sensor: it finds the action that best explains the change between two camera images (a reading). It stores every hand it identifies, so a returning set-up is recognised rather than re-learnt. In closed-loop tests, LoH more than doubles success on three simulated tasks, from 28.6% to 62.3%, and loses only 0.6 percentage points when nothing changes. It also works, unchanged, with two other world models and on two of three models pretrained on real robot data, and its gain grows with the quality of its readings.