Unassigned · 2026-10-14

AnyRank: Reliable Anytime Policy Ranking via Sequential Testing

Apurva Badithela, David Snyder, Tenny Yin, Jade Sceats, George Pappas, Nikolai Matni, Masha Itkina, Haruki Nishimura, Anirudha Majumdar

arXiv:2610.14603PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AnyRank is a sequential multi-policy ranking method for robot policy evaluation under limited hardware trials. It adaptively allocates trials to hard-to-separate policies, outputs (partial) rankings at any stopping time with bounded family-wise error rate, and can incorporate prior information (imperfect simulation/world-model/held-out evaluations) to improve efficiency without invalidating statistical assurances.

Why it matters

Robot policy evaluation is costly and often uses small numbers of hardware rollouts, so rankings can be unreliable when many policies are compared. AnyRank aims to provide anytime-valid, statistically reliable policy rankings with family-wise error control, while also reducing hardware evaluations via sequential testing, graphical multi-hypothesis corrections, and active (adaptive) sampling using priors.

Method

  • Formulates multi-policy ranking as a multiple-hypothesis testing problem using sequential pairwise comparisons (N-SCORE) that yield anytime-valid p-values and control Type-I error at a tunable level.
  • Uses graphical hypothesis testing procedures to control family-wise error rate across many pairwise comparisons via risk-budget allocation and reallocation, reconciling batch graphical updates with sequential evaluation.
  • Adds efficient active sampling: selects the next policy to evaluate using hedged/advantaged selection based on current anytime-valid evidence, and can incorporate priors through softmax-style risk allocation and graph weights.
Abstract (from arXiv)

Due to the high cost of hardware experiments, robot policies are predominantly evaluated by comparing their empirical performance on a limited number of hardware trials. However, finite-sample statistical uncertainty can lead to erroneous conclusions, especially when ranking multiple policies using a fixed batch of evaluation data. We propose AnyRank: a sequential method for multi-policy ranking that (i) adaptively allocates trials to hard-to-separate policies, (ii) provides (partial) rankings at any stopping time while bounding the probability of an incorrect ranking, and (iii) optionally exploits prior information to prioritize hardware evaluations more efficiently. This prior information could be informed from expert intuition, a small set of held-out hardware evaluations, or from policy rollouts in a simulator or an action-conditioned video world model. Notably, the prior can be imperfect; while a strong prior can provide efficiency gains on hardware, any inaccuracies in prior policy rankings do not invalidate the statistical assurances of AnyRank. We achieve the aforementioned properties through novel theoretical contributions that combine sequential testing with multiple-hypothesis graphical testing approaches. Experiments in simulation and real-world policy evaluation benchmarks show that AnyRank requires up to 50% fewer hardware evaluations compared to existing statistically rigorous approaches, while guarding against the incorrect rankings produced by heuristic methods used by state-of-the-art benchmarks.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-10 · Indexed by Pub-AI: 2026-10-14 UTC

preprint

AnyRank: Reliable Anytime Policy Ranking via Sequential Testing

Apurva Badithela, David Snyder, Tenny Yin, Jade Sceats, George Pappas, Nikolai Matni, Masha Itkina, Haruki Nishimura, Anirudha Majumdar

Preprint posted: 2026-10-10 · Original record ↗

arxiv
2610.14603
Explicit source versions (1)
  • v2 · 2026-10-16
  1. Preprint available

    Source date: 2026-10-10 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
  2. Explicit source version detected · v2

    Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Due to the high cost of hardware experiments, robot policies are predominantly evaluated by comparing their empirical performance on a limited number of hardware trials. However, finite-sample statistical uncertainty can lead to erroneous conclusions, especially when ranking multiple policies using a fixed batch of evaluation data. We propose AnyRank: a sequential method for multi-policy ranking that (i) adaptively allocates trials to hard-to-separate policies, (ii) provides (partial) rankings at any stopping time while bounding the probability of an incorrect ranking, and (iii) optionally exploits prior information to prioritize hardware evaluations more efficiently. This prior information could be informed from expert intuition, a small set of held-out hardware evaluations, or from policy rollouts in a simulator or an action-conditioned video world model. Notably, the prior can be imperfect; while a strong prior can provide efficiency gains on hardware, any inaccuracies in prior policy rankings do not invalidate the statistical assurances of AnyRank. We achieve the aforementioned properties through novel theoretical contributions that combine sequential testing with multiple-hypothesis graphical testing approaches. Experiments in simulation and real-world policy evaluation benchmarks show that AnyRank requires up to 50% fewer hardware evaluations compared to existing statistically rigorous approaches, while guarding against the incorrect rankings produced by heuristic methods used by state-of-the-art benchmarks.