Driving · 2026-10-14
Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
Ali-Adeeb Abbas, Esmaeil Seraj, Behrad Toghi, Taskin Padir
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors test spatial instruction sensitivity in world-action models by running 864 rollouts across four RoboLab scenes with the same initial states and goals but three equivalent goal wordings (direct, target-first, reference-first). Describing relations via the reference object (RF) reduces reaching stable placement vs target-first (TF) by 12.2 percentage points on average. They also show scoring can credit moving the wrong object and that adding images can disrupt text-only instruction interpretation.
Why it matters
The paper argues that world-action models should preserve the intended manipulation under equivalent spatial descriptions, but aggregate placement rates can miss relation-specific sensitivity and can even reflect incorrect object manipulation. The authors motivate evaluations that check consistency across equivalent instructions, verify which object moved, and test whether the requested arrangement persists to the end of the rollout.
Method
- Run paired comparisons on three released world-action models (Cosmos3 Nano, Cosmos3 Edge, FLUX 3 Action) in four RoboLab scenes, repeating the same physical starts under DIR, TF, and RF instructions (864 episodes).
- Measure stable placement using two rates: any-time stable placement vs stable placement during the final second; compare TF–RF differences while averaging over goals within scenes and weighting scenes equally.
- Diagnose language understanding by testing the vision-language backbone on the same cube/butter/mustard instruction goals, comparing text-only vs images plus camera descriptions.
Limitation
The authors note that the controlled VLM backbone tests do not establish the causes of rollout failures or effects of robot training (Appendix F).
Abstract (from arXiv)
World-action models should preserve the intended manipulation across equivalent descriptions of a spatial goal. We evaluate this consistency in 864 rollouts from three models across four RoboLab scenes, holding initial states and task goals fixed. Describing relations through the reference object rather than the object-to-move lowers the rate of reaching a stable placement by 12.2 percentage points on average. The effect of instruction rephrasing varies across spatial relations, so aggregate scores can hide sensitivity to wording. Standalone tests of the policies' language understanding identify relation reversals and show that adding images can disrupt otherwise correct text-only interpretations. Rollout analysis also distinguishes spatial placement from instruction following: moving the wrong object can satisfy the requested relation, and rates of stable placement fall from 43.2% at any time to 21.8% at episode end. These findings motivate a more thorough evaluation of world-action models' steerability, language understanding, and grounding. Evaluations should test consistency across equivalent instructions, verify that the intended object moved, and check whether the requested arrangement persists to the end of the rollout.
Related papers
- Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
- Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
- A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding
- Keeping JEPA World Models Plannable When Little of the Frame Moves
- Causal Analysis of Robot Foundation Models Enables Efficiency Gains
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-14 UTC
Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
Preprint posted: 2026-10-12 · Original record ↗
- arxiv
- 2610.16242
Explicit source versions (1)
- v3 · 2026-10-15
- Explicit source version detected · v3
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World-action models should preserve the intended manipulation across equivalent descriptions of a spatial goal. We evaluate this consistency in 864 rollouts from three models across four RoboLab scenes, holding initial states and task goals fixed. Describing relations through the reference object rather than the object-to-move lowers the rate of reaching a stable placement by 12.2 percentage points on average. The effect of instruction rephrasing varies across spatial relations, so aggregate scores can hide sensitivity to wording. Standalone tests of the policies' language understanding identify relation reversals and show that adding images can disrupt otherwise correct text-only interpretations. Rollout analysis also distinguishes spatial placement from instruction following: moving the wrong object can satisfy the requested relation, and rates of stable placement fall from 43.2% at any time to 21.8% at episode end. These findings motivate a more thorough evaluation of world-action models' steerability, language understanding, and grounding. Evaluations should test consistency across equivalent instructions, verify that the intended object moved, and check whether the requested arrangement persists to the end of the rollout.