Driving · 2026-11-10

AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

Shrey Singhal, Nannan Huang, Haytham M. Fayek

arXiv:2611.10750PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AERO is a fully online RLHF algorithm that routes oracle preference feedback only to contexts with high ensemble disagreement (vote entropy) and uses ensemble-based synthetic labels elsewhere. It uses a deep ensemble Epistemic Reward Model (ERM) for both epistemic exploration and synthetic labeling, with adaptive batch-level Otsu thresholding and pair-level filtering. The authors report lower oracle-query budgets than online DPO and SEA while keeping competitive alignment performance.

Why it matters

RLHF pipelines are often bottlenecked by preference-label cost. AERO targets this by concentrating expensive oracle queries on the most informative contexts, using committee disagreement to estimate where oracle supervision matters and synthetic labeling where the ensemble is more consistent. The paper also reports audit results for synthetic-label reliability and resource-accounting comparisons in terms of oracle usage, GPU-hours, and win-rate.

Method

  • Maintain an ensemble-based Epistemic Reward Model (ERM) whose heads model epistemic uncertainty; use vote entropy (committee disagreement) as the routing score.
  • Route each online batch: send high vote-entropy contexts to the oracle and low vote-entropy contexts to ERM synthetic preference labeling; choose the routing threshold via Otsu’s method.
  • Use branch-specific dueling pair selection and aleatoric-uncertainty filtering (exclude pairs with mean binary entropy above α=0.6); train policy on both oracle and ERM-labeled data, while ERM trains only on oracle labels.

Limitation

The oracle is a deterministic scalar reward model and does not capture stochasticity of real human preference judgments.

Abstract (from arXiv)

Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

Shrey Singhal, Nannan Huang, Haytham M. Fayek

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.10750
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.10750v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.