Driving · 2026-11-03

World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits

Ahanaf Hasan Ariq, Hasaan Ahmad

arXiv:2611.01139PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors derive a relative simulation identity/bound showing when shared world-model errors can preserve correct pairwise policy rankings even if absolute value estimates are wrong. They propose RAVEL, a selective policy tournament that orders candidates via model rollouts but certifies eliminations only with held-out real audits. In DMC cheetah-run and walker-walk at horizon 500, rankings stay strong (Kendall τ 0.962/0.800) while matched audit errors are weakly correlated (0.007/0.090).

Why it matters

World-model evaluations often report global agreement (e.g., correlations), but policy selection is an argmax problem where wrong absolute calibration can still allow correct ranking. The paper separates (1) ranking fidelity, (2) audit utility from error covariance, and (3) exploitation risks. This helps determine when limited real interaction can reliably certify which policy is best, and when model rankings are insufficient for audit savings.

Method

  • Derive an exact pairwise simulation identity and a relative simulation bound separating common-mode (shared) Bellman residuals from policy-specific error, yielding ranking/top-1 regret certificates.
  • Propose RAVEL: order policies by cheap model rollouts; audit only consequential champion–challenger pairs on held-out real outcomes; eliminate only when real-data lower confidence bounds are positive.
  • Conduct a controlled DMC study (cheetah-run, walker-walk) comparing world-model tournament variants (RAVEL-P/RAVEL-U) against real-only successive halving across horizons and budgets, plus a frozen-model stress test.

Limitation

The authors state that certificates require representative audits, bounded errors, and finite-rollout uncertainty; RAVEL cannot remove support mismatch, and they only study two state-based tasks, one horizon-500 suite, overlapping pools from 45 policies, and estimated 50-episode references (conditional rather than independent-run generalization).

Abstract (from arXiv)

Selecting a reinforcement-learning (RL) policy usually requires additional environment interaction, while learned world models offer cheap but potentially biased evaluations. This raises two distinct questions: can shared model errors preserve pairwise rankings despite poor absolute values, and do they make a small number of real audits more informative? We derive an exact pairwise simulation identity and a relative bound separating common-mode Bellman residuals from policy-specific error. The analysis motivates RAVEL (Relative Auditing of Value Errors in Learned models), a selective policy tournament whose eliminations use held-out real outcomes. We test 45 SAC, TD3, and PPO policies per task in DMC cheetah-run and walker-walk. At rollout horizon 500, model rankings remain strong (Kendall tau = 0.962 and 0.800), and same-run differential-error ratios are 0.674 and 0.106. Yet mean paired audit-error correlations are only 0.007 and 0.090, and pairing changes standard deviation by -2.5 percent and +1.9 percent. Simultaneous intervals attain 99.98 percent and 99.75 percent coverage, but certify only 13.2 percent and 0 percent of decisions at eight audits. RAVEL improves low-budget walker-walk regret, whereas real-only successive halving is better at the largest budget, rejecting the precommitted twofold efficiency target. Frozen-model optimization lowers real return without increasing predicted return on average. Thus ranking fidelity, useful audit covariance, and upward model exploitation are distinct properties, separated in part by closed-loop trajectory divergence.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits

Ahanaf Hasan Ariq, Hasaan Ahmad

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.01139
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.01139v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Selecting a reinforcement-learning (RL) policy usually requires additional environment interaction, while learned world models offer cheap but potentially biased evaluations. This raises two distinct questions: can shared model errors preserve pairwise rankings despite poor absolute values, and do they make a small number of real audits more informative? We derive an exact pairwise simulation identity and a relative bound separating common-mode Bellman residuals from policy-specific error. The analysis motivates RAVEL (Relative Auditing of Value Errors in Learned models), a selective policy tournament whose eliminations use held-out real outcomes. We test 45 SAC, TD3, and PPO policies per task in DMC cheetah-run and walker-walk. At rollout horizon 500, model rankings remain strong (Kendall tau = 0.962 and 0.800), and same-run differential-error ratios are 0.674 and 0.106. Yet mean paired audit-error correlations are only 0.007 and 0.090, and pairing changes standard deviation by -2.5 percent and +1.9 percent. Simultaneous intervals attain 99.98 percent and 99.75 percent coverage, but certify only 13.2 percent and 0 percent of decisions at eight audits. RAVEL improves low-budget walker-walk regret, whereas real-only successive halving is better at the largest budget, rejecting the precommitted twofold efficiency target. Frozen-model optimization lowers real return without increasing predicted return on average. Thus ranking fidelity, useful audit covariance, and upward model exploitation are distinct properties, separated in part by closed-loop trajectory divergence.