Regime-Aware Judge Auditing via Separated Denominators and Tie Calibration
Use a regime-aware benchmark/audit protocol that evaluates LLM judges on multiple, explicitly separated response regimes—e.g., verifiable constraint-violation evidence, preference judgments with “four-of-five” denominators, and within-panel disagreement—rather than collapsing everything into a single score. Add tie-band calibration and measure call-to-call variation so judge reliability is assessed across rubric regimes.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Judge reliability can be confounded when evaluations report only a single scalar score per example. Separating regimes and explicitly calibrating tie handling helps distinguish failures due to rubric design, evidence extraction, preference stability, and ambiguity that reflects legitimate human disagreement.
What changed
Compared to one-shot, single-score evaluation, the protocol (as described) (1) assigns separate denominators/evidence rules for constraint violations, preference preference-threshold outcomes (four-of-five), and within-panel disagreement; and (2) introduces tie-band calibration and audit of judge-call variation, including analysis of vote-margin stability.
Where it helps
- Auditing LLM-based evaluators for tasks with mixed objective constraints and subjective preferences
- Benchmark design that separates rubric regimes instead of using a single score
- Evaluating judge stability across repeated calls and calibrated tie bands
- Human-rater agreement and vote-margin analysis for adjudication policies
- Designing audit sets with verifiable interventions and blinded multi-rater records
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The described method is presented for a specific setting (a Chinese multi-episode drama auditing task) and relies on the availability of verifiable constraints, designed interventions, and multi-rater (panel) human vote records for regime definitions and tie calibration.
Sources
- DramaGrow: A Regime-Aware Benchmark for Auditing LLM Judges of Multi-Episode Drama ↗
arXiv · primary · Checked 2026-11-10
