Driving · 2026-10-20
LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models
Xueyang Zhou, Yangming Xu, Qianjiang Li, Guiyao Tie, Junhao Wu, Ziheng Huang, Guowen Zhang, Zhijie Yuan, Di Wu, Duanfeng Chu, Pan Zhou, Lichao Sun, Yongchao Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
LIBERO-Pro2.0 is a robustness benchmark for embodied foundation models under distribution shifts. It adds 22 static perturbations and 20 runtime interventions across six dimensions, yielding 1,648 validated shifted task instances, plus 39,390 training trajectories from model-generated and human demonstrations with six rounds of human review. The authors evaluate 17 policies and report large robustness gaps under runtime shifts.
Why it matters
The authors argue that strong nominal success on LIBERO may not reflect robust embodied capabilities and that robustness needs evaluation under both static shifts and runtime interventions. LIBERO-Pro2.0 provides a systematic, validated suite for testing robustness under distribution shifts, and their results quantify performance degradation and failures to generalize or recover when conditions change during execution.
Method
- Introduces LIBERO-Pro2.0: 22 static distribution shifts and 20 runtime interventions across six dimensions (Condition, Environment, Observation, Execution, Robot State Configuration, Language), yielding 1,648 validated shifted task instances.
- Collects 39,390 training trajectories via model-generated and human demonstrations, with shifted task instances undergoing six rounds of iterative human review and correction for physical plausibility and task validity.
- Evaluates 17 representative robot policies (vision-language-action, world-action, robustness-oriented) on 8 held-out evaluation tasks under nominal execution, 22 static shifts, and 20 runtime shifts.
Abstract (from arXiv)
LIBERO has emerged as a widely adopted benchmark for evaluating robotic manipulation policies. However, strong performance on LIBERO does not necessarily indicate robust embodied capabilities, as models may exploit benchmark-specific regularities rather than generalize to diverse operating conditions. In this work, we introduce LIBERO-Pro2.0, a comprehensive benchmark for systematically evaluating the robustness of embodied foundation models under distribution shifts. LIBERO-Pro2.0 encompasses 22 static perturbations and 20 runtime interventions across six dimensions: Condition, Environment, Observation, Execution, Robot State Configuration, and Language, yielding 1,648 validated shifted task instances. To support robustness-oriented learning, we further collect 39,390 training trajectories through model-generated and human demonstrations. All shifted task instances undergo six rounds of iterative human review and correction to ensure physical plausibility and task validity. We evaluate 17 representative robot policies spanning vision-language-action models, world-action models, and robustness-oriented policies. Our results reveal substantial robustness gaps across all evaluated paradigms: under runtime distribution shifts, the evaluated models exhibit an average performance drop of 26.5 percentage points, with individual drops reaching 32.5 points. These findings highlight the gap between nominal benchmark performance and robustness under distribution shifts, underscoring the need for systematic robustness evaluation of embodied foundation models.
Related papers
- RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence
- Learning Robotic World Models with Semantic and Visual Feedback
- RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
- Vision-Language-Delta-Action: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation
- ProWorld: Multi-View World Model with Proprioceptive Feedback for Closed-Loop Policy Evaluation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-17 · Indexed by Pub-AI: 2026-10-20 UTC
LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models
Preprint posted: 2026-10-17 · Original record ↗
- arxiv
- 2610.24587
Explicit source versions (1)
- v1 · 2026-10-17
- Preprint available
Source date: 2026-10-17 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
LIBERO has emerged as a widely adopted benchmark for evaluating robotic manipulation policies. However, strong performance on LIBERO does not necessarily indicate robust embodied capabilities, as models may exploit benchmark-specific regularities rather than generalize to diverse operating conditions. In this work, we introduce LIBERO-Pro2.0, a comprehensive benchmark for systematically evaluating the robustness of embodied foundation models under distribution shifts. LIBERO-Pro2.0 encompasses 22 static perturbations and 20 runtime interventions across six dimensions: Condition, Environment, Observation, Execution, Robot State Configuration, and Language, yielding 1,648 validated shifted task instances. To support robustness-oriented learning, we further collect 39,390 training trajectories through model-generated and human demonstrations. All shifted task instances undergo six rounds of iterative human review and correction to ensure physical plausibility and task validity. We evaluate 17 representative robot policies spanning vision-language-action models, world-action models, and robustness-oriented policies. Our results reveal substantial robustness gaps across all evaluated paradigms: under runtime distribution shifts, the evaluated models exhibit an average performance drop of 26.5 percentage points, with individual drops reaching 32.5 points. These findings highlight the gap between nominal benchmark performance and robustness under distribution shifts, underscoring the need for systematic robustness evaluation of embodied foundation models.
