Driving · 2026-10-12
Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
Justin Tien-Smith, Matthew Pidden, Rolandos Alexandros Potamias
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Spa-Bench is a real-robot benchmark for spatial grounding in vision-language-action models. It evaluates six tabletop tasks (object state, relative size, referential description, relational placement, ordinal position, counting) under in-distribution vs out-of-distribution spatial configurations using a withheld-cell split and four diagnostics. On 5 evaluated policies, success drops on OOD spatial configurations, indicating weak robust spatial grounding.
Why it matters
The authors argue that conventional VLA benchmarks entangle spatial grounding with object recognition, language interpretation, and motor execution. Spa-Bench isolates spatial grounding by keeping objects, concepts, and action behaviors familiar while changing only the concept–argument pairing via withheld cells, and by adding direct-manipulation, prompt-paraphrase, cross-task-object, and no-intervention diagnostics to probe the source of failures.
Method
- Introduces Spa-Bench, a real-robot benchmark on the SO-101 arm with six tabletop spatial grounding tasks spanning attribute, relation, and scene structure.
- Implements a controlled ID/OOD split via a withheld-cell design: concepts and objects are present in training, but selected spatial concept–argument pairings are reserved for OOD evaluation.
- Adds four diagnostics (direct-manipulation controls, prompt paraphrases, cross-task-object rollouts, and no-intervention scenes) to test physical execution, wording sensitivity, object-task associations, and action gating.
Abstract (from arXiv)
Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-language pretraining into language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-robot behavior? To investigate this, we introduce Spa-Bench, a real-robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including videos, code, and benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.
Related papers
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives
- A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding
- Causal Analysis of Robot Foundation Models Enables Efficiency Gains
- Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
- MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-12 UTC
Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
Preprint posted: 2026-10-08 · Original record ↗
- arxiv
- 2610.12696
Explicit source versions (1)
- v1 · 2026-10-08
- Preprint available
Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-language pretraining into language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-robot behavior? To investigate this, we introduce Spa-Bench, a real-robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including videos, code, and benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.