Driving · 2026-11-10
AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF
Shrey Singhal, Nannan Huang, Haytham M. Fayek
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AERO is a fully online RLHF algorithm that routes oracle preference feedback only to contexts with high ensemble disagreement (vote entropy) and uses ensemble-based synthetic labels elsewhere. It uses a deep ensemble Epistemic Reward Model (ERM) for both epistemic exploration and synthetic labeling, with adaptive batch-level Otsu thresholding and pair-level filtering. The authors report lower oracle-query budgets than online DPO and SEA while keeping competitive alignment performance.
Why it matters
RLHF pipelines are often bottlenecked by preference-label cost. AERO targets this by concentrating expensive oracle queries on the most informative contexts, using committee disagreement to estimate where oracle supervision matters and synthetic labeling where the ensemble is more consistent. The paper also reports audit results for synthetic-label reliability and resource-accounting comparisons in terms of oracle usage, GPU-hours, and win-rate.
Method
- Maintain an ensemble-based Epistemic Reward Model (ERM) whose heads model epistemic uncertainty; use vote entropy (committee disagreement) as the routing score.
- Route each online batch: send high vote-entropy contexts to the oracle and low vote-entropy contexts to ERM synthetic preference labeling; choose the routing threshold via Otsu’s method.
- Use branch-specific dueling pair selection and aleatoric-uncertainty filtering (exclude pairs with mean binary entropy above α=0.6); train policy on both oracle and ERM-labeled data, while ERM trains only on oracle labels.
Limitation
The oracle is a deterministic scalar reward model and does not capture stochasticity of real human preference judgments.
Abstract (from arXiv)
Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.
Related papers
- Statistical Foundations of Reinforcement Learning: A Non-Asymptotic Perspective
- Can Jev be Your Q or Policy in Reinforcement Learning?
- Recursive State-Augmented Language Models: Persistent Latent-State Adaptation of Pretrained Language Models
- Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
- QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.10750
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.10750v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.
