Latent Dynamics · 2026-10-06
CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery
Tao Feng, Fangxu Yu, Zijie Lei, Jiaru Zou, Changjiang Jiang, Yi Yan, Jiaxuan You, Pan Lu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose CURIO, a curiosity-driven test-time learning framework for open-ended discovery. CURIO trains an Intrinsic Curiosity World Model (ICWM) to model token-level transitions in the policy’s hidden-state representation and adds a prediction-error bonus at sampled tokens outside the policy’s top-k choices. Epoch normalization and an annealed curiosity weight regulate the bonus contribution during test-time RL.
Why it matters
Open-ended discovery needs exploration beyond what task rewards reveal. The authors argue that task-only RL can suppress low-reward yet potentially promising directions, and they introduce a complementary exploration signal learned from the policy’s own representation dynamics. CURIO is evaluated on six mathematical discovery tasks and single-cell denoising, reporting improvements over a matched task-only RL control and higher code-level structural diversity.
Method
- Adds an Intrinsic Curiosity World Model (ICWM) that learns token-level transitions in the policy hidden-state representation and produces prediction-error bonuses.
- Applies the curiosity bonus selectively: it is retained only for sampled tokens outside the policy’s top-k choices (with an epoch-normalized residual).
- Combines the task coefficient with the curiosity bonus in the test-time policy update using an annealed curiosity weight.
Limitation
The authors note that high ICWM prediction error does not establish semantic novelty, insufficient prior exploration, or future task improvement, and it depends on predictor capacity and representation drift; they also state the formulation does not provide an anti-collapse guarantee.
Abstract (from arXiv)
Open-ended discovery requires learning from repeated attempts while continuing to explore directions whose value is not yet apparent. Search with a frozen large language model (LLM) can reuse previous solutions in context, but cannot update the model from its successes and failures on the test problem. Reinforcement learning (RL) enables such adaptation; however, strongly favoring high-reward trajectories may suppress low-reward yet potentially promising directions too early. We introduce CURIO, a curiosity-driven test-time learning framework that complements task feedback with an Intrinsic Curiosity World Model (ICWM). The ICWM learns transitions in the policy's hidden-state representation and supplies prediction-error bonuses at sampled tokens outside the policy's top-k choices. Epoch normalization and an annealed weight regulate their contribution to the policy update. On six mathematical discovery tasks and single-cell denoising with Qwen3 backbones from 8B to 235B, three-run means improve over a matched task-only RL control on five mathematical objectives, match the best reported performance on Circle Packing, and improve denoising Score and mean squared error (MSE) on both held-out corpora at every tested scale. Relative gains reach 18.3% on Hadamard and 10.8% on denoising Score. Code-diversity measurements show greater structural variation among generated programs, supporting curiosity as a complementary exploration signal for learning in open-ended discovery.
Related papers
- When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity
- EvoBelief: Falsifiable World Models for Self-Improving LLM Agents
- JANUS: Enhancing Long-Horizon Autonomous Research with Co-Evolving World Models
- ULTRADISCOVERY: Abductive Exploration in an Interconnected, Epistemically Open Universe
- Building World Models from Intrinsic Motivation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-04 · Indexed by Pub-AI: 2026-10-06 UTC
CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery
Preprint posted: 2026-10-04 · Original record ↗
- arxiv
- 2610.04851
Explicit source versions (1)
- v1 · 2026-10-04
- Preprint available
Source date: 2026-10-04 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Open-ended discovery requires learning from repeated attempts while continuing to explore directions whose value is not yet apparent. Search with a frozen large language model (LLM) can reuse previous solutions in context, but cannot update the model from its successes and failures on the test problem. Reinforcement learning (RL) enables such adaptation; however, strongly favoring high-reward trajectories may suppress low-reward yet potentially promising directions too early. We introduce CURIO, a curiosity-driven test-time learning framework that complements task feedback with an Intrinsic Curiosity World Model (ICWM). The ICWM learns transitions in the policy's hidden-state representation and supplies prediction-error bonuses at sampled tokens outside the policy's top-k choices. Epoch normalization and an annealed weight regulate their contribution to the policy update. On six mathematical discovery tasks and single-cell denoising with Qwen3 backbones from 8B to 235B, three-run means improve over a matched task-only RL control on five mathematical objectives, match the best reported performance on Circle Packing, and improve denoising Score and mean squared error (MSE) on both held-out corpora at every tested scale. Relative gains reach 18.3% on Hadamard and 10.8% on denoising Score. Code-diversity measurements show greater structural variation among generated programs, supporting curiosity as a complementary exploration signal for learning in open-ended discovery.
