Driving · 2026-10-09
World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
Junwei Quan, Evgenii Opryshko, Nicholas Rhinehart, Igor Gilitschenski
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose World-Model Policy Arbiter (WMPA), a test-time framework that arbitrates among a bank of frozen goal-conditioned policies. WMPA rolls out each candidate in a learned state-space world model, scores imagined futures with a shared goal-conditioned value function, and executes the highest-scoring policy for a commitment interval before re-arbitrating. On 18 OGBench state-based datasets, it improves macro-average success from 44% to 58%.
Why it matters
Offline goal-conditioned RL often yields multiple competent but complementary policies, yet no single policy is best across environments, goals, and task phases. WMPA addresses policy arbitration when value functions are not comparable across algorithms and frequent switching can destabilize control, by comparing policies through predicted consequences in a learned world model and using shared value scoring.
Method
- Learn an ensemble of state-space dynamics models in the environment state space and use them to imagine k steps of each frozen policy’s closed-loop behavior.
- Score all imagined futures with a single shared goal-conditioned value function so different bank members can be compared on a common scale.
- Select the highest-scoring policy and execute it for c environment steps (commitment interval), then repeat arbitration to limit excessive switching.
Limitation
WMPA is restricted to selecting behaviors represented in the policy bank (it can only select behaviors already present in the bank).
Abstract (from arXiv)
Offline goal-conditioned reinforcement learning (GCRL) has produced a diverse set of goal-reaching algorithms, yet no single algorithm performs best across environments, goals, and even different phases of the same task. Rather than deploying only the best-performing policy, we ask whether a set of frozen goal-conditioned policies can be used collectively as a portfolio, deciding at every state which policy should act. Choosing a policy at each state is not straightforward. The policies' own value functions cannot be compared directly: they may use different scales, and some policies have no value function. We need to judge each policy by the states it is likely to reach, even though we can execute only one policy at a time. We also need to avoid switching so often that control becomes unstable. To address these challenges, we introduce World-Model Policy Arbiter (WMPA), a test-time framework that, given a set of frozen policies as input, rolls out each frozen policy in a learned state-space world model, evaluates the imagined futures with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval before the next round of arbitration (policy selection). WMPA assumes access to a bank of frozen goal-conditioned policies and requires neither policy retraining nor privileged task-specific knowledge. Under the official OGBench evaluation protocol on 18 state-based datasets spanning maze navigation as well as cube, scene, and puzzle manipulation, WMPA improves the macro-average success rate from the 44% achieved by the best policy selected per dataset to 58%, with statistically significant gains on 12 datasets. These gains include +33 percentage points on cube-double-play and +36 percentage points on scene-play.
Related papers
- Learning Goal-Conditioned Subgoals through Frozen Hierarchical World Models
- VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching
- DRILL: Training World Models to Improve Policies, Not Predict Pixels
- Imagining Goals with Geometric Horizon Models for Policy Generalization
- Back on Track: Deployment-time Policy Improvement via World Model Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-09 UTC
World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.10932
Explicit source versions (1)
- v1 · 2026-10-07
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Offline goal-conditioned reinforcement learning (GCRL) has produced a diverse set of goal-reaching algorithms, yet no single algorithm performs best across environments, goals, and even different phases of the same task. Rather than deploying only the best-performing policy, we ask whether a set of frozen goal-conditioned policies can be used collectively as a portfolio, deciding at every state which policy should act. Choosing a policy at each state is not straightforward. The policies' own value functions cannot be compared directly: they may use different scales, and some policies have no value function. We need to judge each policy by the states it is likely to reach, even though we can execute only one policy at a time. We also need to avoid switching so often that control becomes unstable. To address these challenges, we introduce World-Model Policy Arbiter (WMPA), a test-time framework that, given a set of frozen policies as input, rolls out each frozen policy in a learned state-space world model, evaluates the imagined futures with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval before the next round of arbitration (policy selection). WMPA assumes access to a bank of frozen goal-conditioned policies and requires neither policy retraining nor privileged task-specific knowledge. Under the official OGBench evaluation protocol on 18 state-based datasets spanning maze navigation as well as cube, scene, and puzzle manipulation, WMPA improves the macro-average success rate from the 44% achieved by the best policy selected per dataset to 58%, with statistically significant gains on 12 datasets. These gains include +33 percentage points on cube-double-play and +36 percentage points on scene-play.