Driving · 2026-10-14
A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding
Abdollah Pourasghari, Saeedeh Sadat Sadidpour, Mohammad Roustaei
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce the first Persian vision-language-action dataset for instruction-to-plan grounding that is ambiguity- and infeasibility-aware. It contains 258 instances over 230 real household/office photographs with reference plans plus ambiguity/infeasibility labels and disambiguating QA; training includes auto-generated chain-of-thought rationales. The authors ablate retrieval, CoT prompting, and disambiguation, introduce PGOS-FA, and report that retrieval-augmentation is a dominant statistically significant effect.
Why it matters
The paper targets a gap in embodied instruction-to-plan benchmarks: prior resources are mostly English and often assume instructions are clear and executable, while real instructions can be ambiguous or physically infeasible. A Persian, single-image dataset with explicit ambiguity/infeasibility supervision and an evaluation metric designed for step-order and over-generation penalties enables more reliable study in a low-resource language.
Method
- Dataset: 258 Persian instruction–image instances (230 real household/office photos) with reference action plans, explicit ambiguity and infeasibility labels, and disambiguating QA pairs; training split includes auto-generated chain-of-thought rationales.
- Evaluation: ablate retrieval-augmentation, chain-of-thought prompting, and an agentic disambiguation step; compare PGOS-FA (order-sensitive, over-generation-penalizing extension of PG2S with Persian embeddings) against an LLM-as-judge protocol.
- Generalization checks: re-run across backbones (open-weight and a stronger closed planner) using matched-generation control and rank-stability testing.
Limitation
Dataset scale is a binding constraint: with 258 total instances, only the single largest effect (retrieval vs. no retrieval) reaches statistical significance; finer-grained component contributions (disambiguation, CoT, prompt language) are directionally consistent but not distinguishable from noise at n = 39.
Abstract (from arXiv)
Converting natural-language instructions into executable action plans conditioned on a scene image is a core capability for embodied human-robot interaction, but existing benchmarks target English, simulated or video-grounded environments, and largely assume instructions are clear and executable. We introduce the first Persian vision-language-action dataset for this task: 258 instances over 230 real household/office photographs, annotated with reference plans, deliberate ambiguity and infeasibility labels, disambiguating question-answer pairs, and an auto-generated chain-of-thought rationale for training. Using this resource, we ablate retrieval-augmentation, chain-of-thought prompting, and an agentic disambiguation step, evaluated with an LLM-as-judge protocol and a new metric, PGOS-FA, extending prior work (PG2S) with step-order sensitivity, an over-generation penalty, and Persian-adapted embeddings. Paired statistical testing across 36 comparisons shows retrieval-augmentation is a large, robustly significant effect, while finer-grained additions are directionally consistent but not statistically distinguishable from noise at this scale, and the LLM-judge metric has near-zero power where PGOS-FA succeeds. A blind human-correlation study on 79 real outputs shows PGOS-FA tracks human judgment at least as closely as the LLM-judge. Testing generalization beyond the ablation's single backbone, we find that most of the gap between an open-weight and a substantially stronger closed planner is attributable to open-weight model generation rather than to the closed model's capability specifically, and that our best configuration significantly outperforms an independently reproduced external baseline despite a weaker backbone. The dataset, PGOS-FA implementation, and evaluation code will be made available upon reasonable request.
Related papers
- UniWALM: Unified World Action Language Model for Embodied Robot Control
- Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
- UniWAM: Unified World-Action Model
- Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
- LAW: Unifying Language, Action, and World Modeling for Agentic Physical Intelligence
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC
A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding
Preprint posted: 2026-10-11 · Original record ↗
- arxiv
- 2610.15666
Explicit source versions (1)
- v1 · 2026-10-11
- Preprint available
Source date: 2026-10-11 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Converting natural-language instructions into executable action plans conditioned on a scene image is a core capability for embodied human-robot interaction, but existing benchmarks target English, simulated or video-grounded environments, and largely assume instructions are clear and executable. We introduce the first Persian vision-language-action dataset for this task: 258 instances over 230 real household/office photographs, annotated with reference plans, deliberate ambiguity and infeasibility labels, disambiguating question-answer pairs, and an auto-generated chain-of-thought rationale for training. Using this resource, we ablate retrieval-augmentation, chain-of-thought prompting, and an agentic disambiguation step, evaluated with an LLM-as-judge protocol and a new metric, PGOS-FA, extending prior work (PG2S) with step-order sensitivity, an over-generation penalty, and Persian-adapted embeddings. Paired statistical testing across 36 comparisons shows retrieval-augmentation is a large, robustly significant effect, while finer-grained additions are directionally consistent but not statistically distinguishable from noise at this scale, and the LLM-judge metric has near-zero power where PGOS-FA succeeds. A blind human-correlation study on 79 real outputs shows PGOS-FA tracks human judgment at least as closely as the LLM-judge. Testing generalization beyond the ablation's single backbone, we find that most of the gap between an open-weight and a substantially stronger closed planner is attributable to open-weight model generation rather than to the closed model's capability specifically, and that our best configuration significantly outperforms an independently reproduced external baseline despite a weaker backbone. The dataset, PGOS-FA implementation, and evaluation code will be made available upon reasonable request.