Driving · 2026-10-02
Kepler: Auditable World Models for ARC-AGI-3
Wensen Wu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Kepler is an open-source harness for ARC-AGI-3 that represents hypotheses as executable world models and validates them via retrospective transition checks and conditional prediction checks. With one frozen Claude Opus 5 configuration, it reports server-verified 100.00 RHAE on all 25 public games. The authors also report several evaluation failures, including harness reconstruction in a control and outcome-vs-tool integrity gaps.
Why it matters
The paper argues that outcome-only ARC scoring can hide how solutions were discovered or whether trajectories respected the intended evaluation protocol. Kepler’s verification-oriented harness separates discovery, final-attempt execution, and server replay, and logs conditional prediction mismatches to support auditable evidence rather than relying solely on final scores.
Method
- Open-source ARC-AGI-3 harness: hypotheses are executable world models; tools record transitions and enable retrospective transition checks.
- Guarded channel: commit.py attempts conditional predictions before each non-reset action; prediction mismatches void the remainder of the plan and record the counterexample.
- Release uses one frozen configuration across games and mechanically executes per-level programs (cleanrun.py) for certified replay; verification includes coordinate checks and halts on detected mismatches or daemon errors.
Limitation
The authors state that whether Kepler’s harness causes the scores versus the models alone is unmeasured until a boundary-correct ablation is run; also, the public set was both development material and evaluation surface, so results establish neither held-out generalization nor private-set performance.
Abstract (from arXiv)
ARC-AGI-3 evaluates agents in interactive environments whose rules and objectives must be inferred from observation. We present Kepler, an open-source harness that represents hypotheses as executable world models and validates them through retrospective transition checks and conditional prediction checks. Under one frozen Claude Opus 5 configuration, Kepler obtained a server-verified 100.00 RHAE on all 25 public games, with no per-game model selection or score-conditioned reruns. On 181 of 183 completed levels, the final Opus attempt used no more actions than the corresponding median-human baseline. The retained board runs used 8,256 environment actions, of which 7,292 occurred in scored levels. Retained local provider-session records yield 858.0 million tokens, 97.37% cache reads, and a \$777.72 cost at September 1, 2026 API list-equivalent rates. We also report three evaluation failures: source-code leakage that produced an invalid perfect run, agents reconstructing a removed harness in a control condition, and autonomous repair masking a broken planner. A single-game observation case study showed that animation frames contained task-relevant information absent from settled text grids. Across the final Claude Opus 5 and GPT-5.6 Sol boards, 48 of 50 game-model cells reached 100. These results indicate that public-set score alone has limited discriminative value and motivate first-attempt, cost-conditioned, and verification-aware reporting.
Related papers
- Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
- Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
- World Editing: Intervening on Executable Worlds at Increasing Depth
- Safe, Persistent, and Evolving Agent Harness for Understanding Partially Observable Worlds
- DashVMC: Real-Time Discrete World Model Control in Geometry Dash
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
Kepler: Auditable World Models for ARC-AGI-3
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00834
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
ARC-AGI-3 evaluates agents in interactive environments whose rules and objectives must be inferred from observation. We present Kepler, an open-source harness that represents hypotheses as executable world models and validates them through retrospective transition checks and conditional prediction checks. Under one frozen Claude Opus 5 configuration, Kepler obtained a server-verified 100.00 RHAE on all 25 public games, with no per-game model selection or score-conditioned reruns. On 181 of 183 completed levels, the final Opus attempt used no more actions than the corresponding median-human baseline. The retained board runs used 8,256 environment actions, of which 7,292 occurred in scored levels. Retained local provider-session records yield 858.0 million tokens, 97.37% cache reads, and a \$777.72 cost at September 1, 2026 API list-equivalent rates. We also report three evaluation failures: source-code leakage that produced an invalid perfect run, agents reconstructing a removed harness in a control condition, and autonomous repair masking a broken planner. A single-game observation case study showed that animation frames contained task-relevant information absent from settled text grids. Across the final Claude Opus 5 and GPT-5.6 Sol boards, 48 of 50 game-model cells reached 100. These results indicate that public-set score alone has limited discriminative value and motivate first-attempt, cost-conditioned, and verification-aware reporting.