Driving · 2026-10-30

RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence

Xin Ma, Furong Chen, Xinlong Man, Bo Zhang, Yalong Guo, Haolan Lei, Chaochen Cui, Hongyuan Zhang, Mulin Chen, Yi Wang, Lap-pui Chau, Chuang Yang

arXiv:2610.40026PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose RoboSeed, a multidimensional evaluation framework for vision-language-action (VLA) and world-action models (WAM). It scores models on five 0–100 dimensions (capability, training affordability, deployment affordability, locally verified research value, reproduction evidence) using fixed normalization anchors, explicit missing-data rules, and audit criteria. They apply it to 12 checkpoint units across VLA/WAM families with reproducible scoring code and machine-readable evidence, but with incomplete deployment measurement in the current snapshot.

Why it matters

Success rate alone can be insufficient for selecting robot action models across different compute budgets and goals. RoboSeed connects task outcomes to resource requirements and to what evidence can be reproduced or audited, while preventing misleading aggregate rankings by withholding incomplete composites. The framework aims to support budget-specific model selection and identify measurement gaps via weighted input coverage.

Method

  • Score each candidate checkpoint on five 0–100 dimensions (capability, downstream training affordability, deployment affordability, locally verified research value, reproduction evidence) using fixed normalization anchors and balanced/training-constrained/deployment-constrained preference settings.
  • Use interface and protocol checks for comparison appropriateness; keep missing measurements explicit as dashes and withhold incomplete composite scores, while using weighted input coverage to identify evidence gaps.
  • Provide reproducible scoring code, machine-readable profiles, source manifests, training observations, task-level outcomes, and an artifact protocol intended to connect selection conclusions to specific experiments.

Limitation

Deployment measurements remain incomplete in this snapshot, and incomplete serving traces and heterogeneous protocols preclude overall numerical rankings.

Abstract (from arXiv)

Choosing a robot action model requires evidence about task capability, adaptation cost, deployment requirements, and the reliability of reported conclusions. Success rates alone offer limited guidance to researchers with different computing budgets and goals. We propose RoboSeed, a multidimensional evaluation framework for vision-language-action (VLA) and world-action models (WAM), within a continuing program to train, evaluate, and openly release robot models with verifiable evidence for model selection. The framework defines five dimensions: capability, downstream training affordability, deployment affordability, locally verified research value, and reproduction evidence. It maps empirical measurements and audit criteria to 0-100 profiles using fixed normalization anchors, with balanced, training-constrained, and deployment-constrained preference settings. Interface and protocol checks establish when comparisons are appropriate. Missing measurements remain explicit, incomplete aggregate scores are withheld, and weighted input coverage identifies evidence gaps. We apply the framework to representative VLA and WAM models, drawing on local training, released-weight evaluation, task-level audits, and an evaluation-system study. The results reveal budget-dependent backbone choices, sensitivity to adaptation recipes, and differences between language and visual task interfaces. Reproducible scoring code and machine-readable evidence support extension to additional models. The study provides dimension-level assessments and priorities for further measurement, while incomplete serving traces and heterogeneous protocols preclude overall numerical rankings.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-10-30 · Indexed by Pub-AI: 2026-10-30 UTC

preprint

RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence

Xin Ma, Furong Chen, Xinlong Man, Bo Zhang, Yalong Guo, Haolan Lei, Chaochen Cui, Hongyuan Zhang, Mulin Chen, Yi Wang, Lap-pui Chau, Chuang Yang

First announced by arXiv: 2026-10-30 · Original record ↗

arxiv
2610.40026
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-10-30 · First detected: 2026-10-30 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40026v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Choosing a robot action model requires evidence about task capability, adaptation cost, deployment requirements, and the reliability of reported conclusions. Success rates alone offer limited guidance to researchers with different computing budgets and goals. We propose RoboSeed, a multidimensional evaluation framework for vision-language-action (VLA) and world-action models (WAM), within a continuing program to train, evaluate, and openly release robot models with verifiable evidence for model selection. The framework defines five dimensions: capability, downstream training affordability, deployment affordability, locally verified research value, and reproduction evidence. It maps empirical measurements and audit criteria to 0-100 profiles using fixed normalization anchors, with balanced, training-constrained, and deployment-constrained preference settings. Interface and protocol checks establish when comparisons are appropriate. Missing measurements remain explicit, incomplete aggregate scores are withheld, and weighted input coverage identifies evidence gaps. We apply the framework to representative VLA and WAM models, drawing on local training, released-weight evaluation, task-level audits, and an evaluation-system study. The results reveal budget-dependent backbone choices, sensitivity to adaptation recipes, and differences between language and visual task interfaces. Reproducible scoring code and machine-readable evidence support extension to additional models. The study provides dimension-level assessments and priorities for further measurement, while incomplete serving traces and heterogeneous protocols preclude overall numerical rankings.