Unassigned · 2026-10-14
AnyRank: Reliable Anytime Policy Ranking via Sequential Testing
Apurva Badithela, David Snyder, Tenny Yin, Jade Sceats, George Pappas, Nikolai Matni, Masha Itkina, Haruki Nishimura, Anirudha Majumdar
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AnyRank is a sequential multi-policy ranking method for robot policy evaluation under limited hardware trials. It adaptively allocates trials to hard-to-separate policies, outputs (partial) rankings at any stopping time with bounded family-wise error rate, and can incorporate prior information (imperfect simulation/world-model/held-out evaluations) to improve efficiency without invalidating statistical assurances.
Why it matters
Robot policy evaluation is costly and often uses small numbers of hardware rollouts, so rankings can be unreliable when many policies are compared. AnyRank aims to provide anytime-valid, statistically reliable policy rankings with family-wise error control, while also reducing hardware evaluations via sequential testing, graphical multi-hypothesis corrections, and active (adaptive) sampling using priors.
Method
- Formulates multi-policy ranking as a multiple-hypothesis testing problem using sequential pairwise comparisons (N-SCORE) that yield anytime-valid p-values and control Type-I error at a tunable level.
- Uses graphical hypothesis testing procedures to control family-wise error rate across many pairwise comparisons via risk-budget allocation and reallocation, reconciling batch graphical updates with sequential evaluation.
- Adds efficient active sampling: selects the next policy to evaluate using hedged/advantaged selection based on current anytime-valid evidence, and can incorporate priors through softmax-style risk allocation and graph weights.
Abstract (from arXiv)
Due to the high cost of hardware experiments, robot policies are predominantly evaluated by comparing their empirical performance on a limited number of hardware trials. However, finite-sample statistical uncertainty can lead to erroneous conclusions, especially when ranking multiple policies using a fixed batch of evaluation data. We propose AnyRank: a sequential method for multi-policy ranking that (i) adaptively allocates trials to hard-to-separate policies, (ii) provides (partial) rankings at any stopping time while bounding the probability of an incorrect ranking, and (iii) optionally exploits prior information to prioritize hardware evaluations more efficiently. This prior information could be informed from expert intuition, a small set of held-out hardware evaluations, or from policy rollouts in a simulator or an action-conditioned video world model. Notably, the prior can be imperfect; while a strong prior can provide efficiency gains on hardware, any inaccuracies in prior policy rankings do not invalidate the statistical assurances of AnyRank. We achieve the aforementioned properties through novel theoretical contributions that combine sequential testing with multiple-hypothesis graphical testing approaches. Experiments in simulation and real-world policy evaluation benchmarks show that AnyRank requires up to 50% fewer hardware evaluations compared to existing statistically rigorous approaches, while guarding against the incorrect rankings produced by heuristic methods used by state-of-the-art benchmarks.
Related papers
- Right Ranking, Wrong Scale: Reproducing the Robotic World Model's Dynamics Model and Testing the Uncertainty of Its Follow-up
- Don't Throw Away the Tail: Action Upcycling for Policy Acceleration
- Multi-Objective Adaptation After Deployment: A Position on World Models for Remote Robot Teams
- When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity
- World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-10 · Indexed by Pub-AI: 2026-10-14 UTC
AnyRank: Reliable Anytime Policy Ranking via Sequential Testing
Preprint posted: 2026-10-10 · Original record ↗
- arxiv
- 2610.14603
Explicit source versions (1)
- v2 · 2026-10-16
- Preprint available
Source date: 2026-10-10 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v2
Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
Due to the high cost of hardware experiments, robot policies are predominantly evaluated by comparing their empirical performance on a limited number of hardware trials. However, finite-sample statistical uncertainty can lead to erroneous conclusions, especially when ranking multiple policies using a fixed batch of evaluation data. We propose AnyRank: a sequential method for multi-policy ranking that (i) adaptively allocates trials to hard-to-separate policies, (ii) provides (partial) rankings at any stopping time while bounding the probability of an incorrect ranking, and (iii) optionally exploits prior information to prioritize hardware evaluations more efficiently. This prior information could be informed from expert intuition, a small set of held-out hardware evaluations, or from policy rollouts in a simulator or an action-conditioned video world model. Notably, the prior can be imperfect; while a strong prior can provide efficiency gains on hardware, any inaccuracies in prior policy rankings do not invalidate the statistical assurances of AnyRank. We achieve the aforementioned properties through novel theoretical contributions that combine sequential testing with multiple-hypothesis graphical testing approaches. Experiments in simulation and real-world policy evaluation benchmarks show that AnyRank requires up to 50% fewer hardware evaluations compared to existing statistically rigorous approaches, while guarding against the incorrect rankings produced by heuristic methods used by state-of-the-art benchmarks.
