Driving · 2026-10-09
Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition
Kaisen Yang, Qingle Liu, Kejin Wang, Yicheng Zhao, Jieming Li, Shenghan Zheng, Ruize Yang, Bojun Yang, Heng Gong, Xiang Gao, Lanyue Zhang, Kaiyu Zhong, Zhuo Liu, Shaoxuan Li, Chengxi Li, Yong Yan, Weixuan Zhang, Tianwei Luo, Situ Wang, Youjie Zheng, Sihan Zhao, Shengyuan Wang, Huan-ang Gao, Jiazheng Xu, Xiaohui Xie, Wentao Han, Hongning Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce AAArena, a benchmark for adversarial “agent-for-agent” improvement using adversarial heuristic learning (AHL): AI agents revise executable game policies and supporting software while keeping base model weights fixed. In 12 games with archived human programs, Opus5.5+Claude Code earns 6 gold medals (rank-1 in frozen human ladders). No evaluated configuration tops the remaining 6 ladders; performance is weaker for more complex rule specifications.
Why it matters
The work tests whether learning can be achieved in long-running adversarial competitions by revising executable policies from game feedback, without updating base model weights. By building AAArena and evaluating multiple model–harness configurations under fixed match/evaluation budgets, the authors provide a structured way to study feedback-driven policy improvement, including how opponent selection and replay feedback affect progress and plateaus.
Method
- Defines Adversarial Heuristic Learning (AHL): an AI agent revises an executable policy and supporting development state from game experience/feedback while keeping base model parameters fixed.
- Introduces AAArena: 12 authentic adversarial games with fixed opponent pools, plus budgets for small matches (dense replays) and full-pool evaluations (Elo, rank, outcomes), with retained-champion scoring.
- Evaluates multiple model–harness configurations and tests ablations for opponent selection, dense replay feedback vs win/loss, small-match batch size, and off-policy replay sourcing.
Limitation
The authors report that full-pool evaluations provide feedback during optimization, so the reported results do not measure generalization to unseen opponents.
Abstract (from arXiv)
Adversarial games have driven advances from heuristic search to reinforcement learning, yet learning and adapting strategies from limited samples remain challenging. AI agents offer an alternative by turning game experience into revisions of executable policies. Building on heuristic learning (HL), we formalize Adversarial Heuristic Learning (AHL), a paradigm that uses AI agents as learning engines to refine game policies and supporting software while keeping model weights fixed. We introduce AAArena, a benchmark comprising 12 authentic adversarial games and 1,920 archived human programs, with an evaluation protocol modeled on real-world game competitions. Agents interpret rules, choose opponents, analyze replays, and revise game agents to achieve their highest ranking within fixed match and evaluation budgets. We evaluate \val{completedmodels} model and harness configurations: Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders. Performance is generally weaker in games with more complex rule specifications. Further experiments show that opponent selection and dense feedback support policy improvement, and that agents learn from both on-policy replays of their own matches and off-policy replays of other players' matches. These results highlight HL's potential in adversarial games and identify persistent challenges in game understanding, strategy implementation, and long-horizon policy development.
Related papers
- Player-1: Learning to Play from Synthetic Experience
- AgentGarten: Code Worlds for Evolving Agents
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
- Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
- FARL: A Fully Automated Pipeline of RL Interface and Solver Design
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC
Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition
Preprint posted: 2026-10-08 · Original record ↗
- arxiv
- 2610.12341
Explicit source versions (1)
- v1 · 2026-10-08
- Preprint available
Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Adversarial games have driven advances from heuristic search to reinforcement learning, yet learning and adapting strategies from limited samples remain challenging. AI agents offer an alternative by turning game experience into revisions of executable policies. Building on heuristic learning (HL), we formalize Adversarial Heuristic Learning (AHL), a paradigm that uses AI agents as learning engines to refine game policies and supporting software while keeping model weights fixed. We introduce AAArena, a benchmark comprising 12 authentic adversarial games and 1,920 archived human programs, with an evaluation protocol modeled on real-world game competitions. Agents interpret rules, choose opponents, analyze replays, and revise game agents to achieve their highest ranking within fixed match and evaluation budgets. We evaluate \val{completedmodels} model and harness configurations: Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders. Performance is generally weaker in games with more complex rule specifications. Further experiments show that opponent selection and dense feedback support policy improvement, and that agents learn from both on-policy replays of their own matches and off-policy replays of other players' matches. These results highlight HL's potential in adversarial games and identify persistent challenges in game understanding, strategy implementation, and long-horizon policy development.