Robotics · 2026-10-06
Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
Shaohan Jiang, Jiahang Cao, Qiduo He, Fengting Deng, Kun Wu, Jingkai Sun, Jiaxu Wang, Qiang Zhang, Qihao Zheng, Chunfeng Song, Ping Luo, Andrew F. Luo
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study scene-preserving instruction interventions where the instructed visible target changes but the scene stays fixed, for vision-language-action policies (VLA) and world-action models (WAMs). They report that even with instruction sensitivity and high linear decodability of the instructed target, evaluated policies predominantly approach and pick the original target named by the scene-associated prior, revealing a representation-to-control grounding gap.
Why it matters
Task success alone can mask whether a robot follows language or relies on scene-task correlations. The paper’s controlled target replacement shows a persistent grounding gap: instructed-target information can be encoded and decodable, yet weak instruction contribution to action generation and misaligned target-token attention prevent reliable target selection.
Method
- Scene-preserving instruction interventions (original vs noun replacement vs random sentence) and valid in-scene target substitutions to hold the scene fixed while changing the instructed target.
- Behavioral evaluation via closed-loop rollouts with committed approach/manipulation attempts (Tend) and task completion (Success), comparing current-instructed vs original-target outcomes.
- Mechanistic analyses: instruction sensitivity via instruction perturbations and a layerwise action lens; target/scene decodability via linear probes; instruction vs visual contribution via attention-based token contribution; attention/representation organization via attention analysis, UMAP, and shared semi-NMF.
Abstract (from arXiv)
Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.
Related papers
- Causal Analysis of Robot Foundation Models Enables Efficiency Gains
- Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
- A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding
- Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
- What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC
Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
Preprint posted: 2026-10-05 · Original record ↗
- arxiv
- 2610.06235
Explicit source versions (1)
- v1 · 2026-10-05
- Preprint available
Source date: 2026-10-05 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.