Driving · 2026-10-28

SO-101 Bench: Accessible Spatial Manipulation

Truman Hickok

arXiv:2610.36420PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SO-101 Bench is a simulated, Isaac Lab digital-twin benchmark for language-conditioned spatial manipulation on a low-cost SO-101 arm. It uses 138 objects with seen/unseen splits and evaluates 916 feasibility-checked episodes plus a shared 3,629-demonstration fine-tuning setup. The authors report the twin preserves sim2real policy rankings (Spearman ρ=1.00), though it overestimates success.

Why it matters

The paper targets a gap: no released SO-101 policy “performs spatially constrained, language-conditioned tasks… out of the box.” SO-101 Bench is designed so leaderboard rankings in simulation carry over to the physical arm, and it evaluates multiple learned policies under matched training data/compute. This supports more reliable sim-based benchmarking for spatial, language-conditioned manipulation on accessible robots.

Method

  • Build an Isaac Lab digital twin of a real SO-101 with 138 household objects and seen/unseen-instance/unseen-class splits (37 unseen-class objects kept private).
  • Define four language-conditioned tasks (scored as five families) with automatic scoring from simulator state, plus ROBUST, PLAN (rotate-to-fit) and MEM (occlusion memory) subsets.
  • Train/fine-tune every learned policy on the same 3,629 simulated demonstrations under equal compute; evaluate in sim and run identical weights on the physical arm for sim2real ranking validation.

Limitation

Simulation overestimates every policy’s success by 9 to 31 percentage points.

Abstract (from arXiv)

Low-cost arms such as the SO-101 have become the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box, and existing SO-101 benchmarks run only on physical hardware with simple tasks and few held-out objects. We introduce SO-101 Bench, a simulated benchmark for language-conditioned spatial manipulation built on an Isaac Lab digital twin of a real SO-101 setup. It has four tasks scored as five families, 138 objects in seen, unseen-instance and unseen-class splits, 916 evaluation episodes with subsets for held-out appearances, rotate-to-fit planning and occlusion memory, and 3,629 simulated demonstrations on which every learned policy is fine-tuned under an equal compute budget. The twin ranks policies as the real robot does: five policies trained without real data, four VLAs and an LLM agent, keep their exact ranking when the same weights run on the physical arm in an unseen environment with unseen objects (Spearman rho = 1.00, MMRV = 0.000), although simulation overestimates success. Across nine VLAs and world-action models, the best learned policies score near 61%, with spatial placement, unseen object classes and rotate-to-fit planning remaining open. An LLM agent that writes code and hands grasping to a learned policy scores 83.2%, and an ablation shows that this lead comes from the choice of harness and the choice of language model.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC

preprint

SO-101 Bench: Accessible Spatial Manipulation

Truman Hickok

Preprint posted: 2026-10-27 · Original record ↗

arxiv
2610.36420
Explicit source versions (1)
  • v1 · 2026-10-27
  1. Preprint available

    Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline

    Source evidence (3)
Abstract · arxiv

Low-cost arms such as the SO-101 have become the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box, and existing SO-101 benchmarks run only on physical hardware with simple tasks and few held-out objects. We introduce SO-101 Bench, a simulated benchmark for language-conditioned spatial manipulation built on an Isaac Lab digital twin of a real SO-101 setup. It has four tasks scored as five families, 138 objects in seen, unseen-instance and unseen-class splits, 916 evaluation episodes with subsets for held-out appearances, rotate-to-fit planning and occlusion memory, and 3,629 simulated demonstrations on which every learned policy is fine-tuned under an equal compute budget. The twin ranks policies as the real robot does: five policies trained without real data, four VLAs and an LLM agent, keep their exact ranking when the same weights run on the physical arm in an unseen environment with unseen objects (Spearman rho = 1.00, MMRV = 0.000), although simulation overestimates success. Across nine VLAs and world-action models, the best learned policies score near 61%, with spatial placement, unseen object classes and rotate-to-fit planning remaining open. An LLM agent that writes code and hands grasping to a learned policy scores 83.2%, and an ablation shows that this lead comes from the choice of harness and the choice of language model.