Unassigned · 2026-11-03
RubricWorld: World Model-Guided Rubric Optimization for LLM Graders
Hongji Pu, Hang Li, Yucheng Chu, Jiliang Tang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RubricWorld is a framework for optimizing LLM grading rubrics without retraining the frozen grader. It trains a world model to predict, from the current rubric state and a proposed edit, correction and degradation rates at ordinal score boundaries using paired before/after grading results. It selects a candidate for evaluation and uses held-out confidence bounds to certify updates. The authors report best results on 20/27 metric comparisons across three benchmarks.
Why it matters
The authors target limitations of existing rubric/prompt optimizers that rely on aggregate validation performance and require rescoring validation answers for each candidate. By learning an edit’s paired effect conditioned on the current rubric state and using certification for acceptance, RubricWorld aims to guide search and reduce the number of grader evaluations needed while accounting for rubric changes across rounds.
Method
- Train a world model on paired grading outcomes (same answers before/after an edit) to predict correction and degradation rates at ordinal score boundaries conditioned on the current rubric state.
- Use the predicted effects to screen and select one candidate for direct evaluation, reducing repeated rescoring; update the model with measured effects even when an edit is rejected.
- Accept rubric updates only when held-out certification answers support target correction and cap new degradation via one-sided finite-sample confidence bounds.
Limitation
The certification procedure addresses one narrow question: whether observed repair and regression rates provide sufficient finite-sample evidence for the prescribed thresholds, and it does not by itself establish educational validity, fairness, distribution-shift robustness, or the absence of all possible future grading errors.
Abstract (from arXiv)
Rubric optimization improves LLM grading by revising scoring instructions while keeping the grader's parameters fixed. Existing optimizers often judge edits by aggregate validation performance, limiting the supervision they retain about which errors an edit corrects or introduces. Testing each candidate also requires rescoring validation answers, which limits how many edits can be explored within a fixed budget. We propose RubricWorld, a framework that learns from previous edit evaluations to guide further search. Its world model predicts an edit's correction and degradation rates at score boundaries, conditioned on the current rubric state. The training signal comes from paired grading results: scores for the same answers before and after an edit. Predictions select one candidate for direct evaluation, and held-out confidence bounds determine whether it corrects enough target errors while keeping new errors within prespecified limits. Measured effects update the model even when the edit is rejected. Across EIR (National), ASAP2.0, and ASAP-SAS with three frozen LLM graders, RubricWorld achieves the highest score in 20 of 27 metric comparisons. On ASAP2.0, it raises gpt-5-mini accuracy from 18.20% with the original rubric to 46.70%, exceeding the strongest baseline, APO, by 22.60 percentage points. code will be released upon acceptance.
Related papers
- One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses
- MLE-World: Predicting Machine Learning Experiment Outcomes with a Learned World Model
- In CEM, a World Model Is Also a Proposal Mechanism
- Predict the Reward, Not the World: World Reward Models for ML Agents
- How To Train Your World Model: Fine-tuning vs RAG for LM-based World Modeling
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
RubricWorld: World Model-Guided Rubric Optimization for LLM Graders
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.01986
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.01986v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Rubric optimization improves LLM grading by revising scoring instructions while keeping the grader's parameters fixed. Existing optimizers often judge edits by aggregate validation performance, limiting the supervision they retain about which errors an edit corrects or introduces. Testing each candidate also requires rescoring validation answers, which limits how many edits can be explored within a fixed budget. We propose RubricWorld, a framework that learns from previous edit evaluations to guide further search. Its world model predicts an edit's correction and degradation rates at score boundaries, conditioned on the current rubric state. The training signal comes from paired grading results: scores for the same answers before and after an edit. Predictions select one candidate for direct evaluation, and held-out confidence bounds determine whether it corrects enough target errors while keeping new errors within prespecified limits. Measured effects update the model even when the edit is rejected. Across EIR (National), ASAP2.0, and ASAP-SAS with three frozen LLM graders, RubricWorld achieves the highest score in 20 of 27 metric comparisons. On ASAP2.0, it raises gpt-5-mini accuracy from 18.20% with the original rubric to 46.70%, exceeding the strongest baseline, APO, by 22.60 percentage points. code will be released upon acceptance.
