Unassigned · 2026-11-03

RubricWorld: World Model-Guided Rubric Optimization for LLM Graders

Hongji Pu, Hang Li, Yucheng Chu, Jiliang Tang

arXiv:2611.01986PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

RubricWorld is a framework for optimizing LLM grading rubrics without retraining the frozen grader. It trains a world model to predict, from the current rubric state and a proposed edit, correction and degradation rates at ordinal score boundaries using paired before/after grading results. It selects a candidate for evaluation and uses held-out confidence bounds to certify updates. The authors report best results on 20/27 metric comparisons across three benchmarks.

Why it matters

The authors target limitations of existing rubric/prompt optimizers that rely on aggregate validation performance and require rescoring validation answers for each candidate. By learning an edit’s paired effect conditioned on the current rubric state and using certification for acceptance, RubricWorld aims to guide search and reduce the number of grader evaluations needed while accounting for rubric changes across rounds.

Method

  • Train a world model on paired grading outcomes (same answers before/after an edit) to predict correction and degradation rates at ordinal score boundaries conditioned on the current rubric state.
  • Use the predicted effects to screen and select one candidate for direct evaluation, reducing repeated rescoring; update the model with measured effects even when an edit is rejected.
  • Accept rubric updates only when held-out certification answers support target correction and cap new degradation via one-sided finite-sample confidence bounds.

Limitation

The certification procedure addresses one narrow question: whether observed repair and regression rates provide sufficient finite-sample evidence for the prescribed thresholds, and it does not by itself establish educational validity, fairness, distribution-shift robustness, or the absence of all possible future grading errors.

Abstract (from arXiv)

Rubric optimization improves LLM grading by revising scoring instructions while keeping the grader's parameters fixed. Existing optimizers often judge edits by aggregate validation performance, limiting the supervision they retain about which errors an edit corrects or introduces. Testing each candidate also requires rescoring validation answers, which limits how many edits can be explored within a fixed budget. We propose RubricWorld, a framework that learns from previous edit evaluations to guide further search. Its world model predicts an edit's correction and degradation rates at score boundaries, conditioned on the current rubric state. The training signal comes from paired grading results: scores for the same answers before and after an edit. Predictions select one candidate for direct evaluation, and held-out confidence bounds determine whether it corrects enough target errors while keeping new errors within prespecified limits. Measured effects update the model even when the edit is rejected. Across EIR (National), ASAP2.0, and ASAP-SAS with three frozen LLM graders, RubricWorld achieves the highest score in 20 of 27 metric comparisons. On ASAP2.0, it raises gpt-5-mini accuracy from 18.20% with the original rubric to 46.70%, exceeding the strongest baseline, APO, by 22.60 percentage points. code will be released upon acceptance.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

RubricWorld: World Model-Guided Rubric Optimization for LLM Graders

Hongji Pu, Hang Li, Yucheng Chu, Jiliang Tang

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.01986
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.01986v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Rubric optimization improves LLM grading by revising scoring instructions while keeping the grader's parameters fixed. Existing optimizers often judge edits by aggregate validation performance, limiting the supervision they retain about which errors an edit corrects or introduces. Testing each candidate also requires rescoring validation answers, which limits how many edits can be explored within a fixed budget. We propose RubricWorld, a framework that learns from previous edit evaluations to guide further search. Its world model predicts an edit's correction and degradation rates at score boundaries, conditioned on the current rubric state. The training signal comes from paired grading results: scores for the same answers before and after an edit. Predictions select one candidate for direct evaluation, and held-out confidence bounds determine whether it corrects enough target errors while keeping new errors within prespecified limits. Measured effects update the model even when the edit is rejected. Across EIR (National), ASAP2.0, and ASAP-SAS with three frozen LLM graders, RubricWorld achieves the highest score in 20 of 27 metric comparisons. On ASAP2.0, it raises gpt-5-mini accuracy from 18.20% with the original rubric to 46.70%, exceeding the strongest baseline, APO, by 22.60 percentage points. code will be released upon acceptance.