Driving · 2026-10-30
RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence
Xin Ma, Furong Chen, Xinlong Man, Bo Zhang, Yalong Guo, Haolan Lei, Chaochen Cui, Hongyuan Zhang, Mulin Chen, Yi Wang, Lap-pui Chau, Chuang Yang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose RoboSeed, a multidimensional evaluation framework for vision-language-action (VLA) and world-action models (WAM). It scores models on five 0–100 dimensions (capability, training affordability, deployment affordability, locally verified research value, reproduction evidence) using fixed normalization anchors, explicit missing-data rules, and audit criteria. They apply it to 12 checkpoint units across VLA/WAM families with reproducible scoring code and machine-readable evidence, but with incomplete deployment measurement in the current snapshot.
Why it matters
Success rate alone can be insufficient for selecting robot action models across different compute budgets and goals. RoboSeed connects task outcomes to resource requirements and to what evidence can be reproduced or audited, while preventing misleading aggregate rankings by withholding incomplete composites. The framework aims to support budget-specific model selection and identify measurement gaps via weighted input coverage.
Method
- Score each candidate checkpoint on five 0–100 dimensions (capability, downstream training affordability, deployment affordability, locally verified research value, reproduction evidence) using fixed normalization anchors and balanced/training-constrained/deployment-constrained preference settings.
- Use interface and protocol checks for comparison appropriateness; keep missing measurements explicit as dashes and withhold incomplete composite scores, while using weighted input coverage to identify evidence gaps.
- Provide reproducible scoring code, machine-readable profiles, source manifests, training observations, task-level outcomes, and an artifact protocol intended to connect selection conclusions to specific experiments.
Limitation
Deployment measurements remain incomplete in this snapshot, and incomplete serving traces and heterogeneous protocols preclude overall numerical rankings.
Abstract (from arXiv)
Choosing a robot action model requires evidence about task capability, adaptation cost, deployment requirements, and the reliability of reported conclusions. Success rates alone offer limited guidance to researchers with different computing budgets and goals. We propose RoboSeed, a multidimensional evaluation framework for vision-language-action (VLA) and world-action models (WAM), within a continuing program to train, evaluate, and openly release robot models with verifiable evidence for model selection. The framework defines five dimensions: capability, downstream training affordability, deployment affordability, locally verified research value, and reproduction evidence. It maps empirical measurements and audit criteria to 0-100 profiles using fixed normalization anchors, with balanced, training-constrained, and deployment-constrained preference settings. Interface and protocol checks establish when comparisons are appropriate. Missing measurements remain explicit, incomplete aggregate scores are withheld, and weighted input coverage identifies evidence gaps. We apply the framework to representative VLA and WAM models, drawing on local training, released-weight evaluation, task-level audits, and an evaluation-system study. The results reveal budget-dependent backbone choices, sensitivity to adaptation recipes, and differences between language and visual task interfaces. Reproducible scoring code and machine-readable evidence support extension to additional models. The study provides dimension-level assessments and priorities for further measurement, while incomplete serving traces and heterogeneous protocols preclude overall numerical rankings.
Related papers
- Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
- Learning Robotic World Models with Semantic and Visual Feedback
- D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models
- A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding
- LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-10-30 · Indexed by Pub-AI: 2026-10-30 UTC
RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence
First announced by arXiv: 2026-10-30 · Original record ↗
- arxiv
- 2610.40026
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-10-30 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40026v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Choosing a robot action model requires evidence about task capability, adaptation cost, deployment requirements, and the reliability of reported conclusions. Success rates alone offer limited guidance to researchers with different computing budgets and goals. We propose RoboSeed, a multidimensional evaluation framework for vision-language-action (VLA) and world-action models (WAM), within a continuing program to train, evaluate, and openly release robot models with verifiable evidence for model selection. The framework defines five dimensions: capability, downstream training affordability, deployment affordability, locally verified research value, and reproduction evidence. It maps empirical measurements and audit criteria to 0-100 profiles using fixed normalization anchors, with balanced, training-constrained, and deployment-constrained preference settings. Interface and protocol checks establish when comparisons are appropriate. Missing measurements remain explicit, incomplete aggregate scores are withheld, and weighted input coverage identifies evidence gaps. We apply the framework to representative VLA and WAM models, drawing on local training, released-weight evaluation, task-level audits, and an evaluation-system study. The results reveal budget-dependent backbone choices, sensitivity to adaptation recipes, and differences between language and visual task interfaces. Reproducible scoring code and machine-readable evidence support extension to additional models. The study provides dimension-level assessments and priorities for further measurement, while incomplete serving traces and heterogeneous protocols preclude overall numerical rankings.
