Latent Dynamics · 2026-10-06
DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents
Qinghua Xu, Guancheng Wang, Boxi Yu, Liting Lin, Lionel Briand
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DreamTest is a world-model surrogate for search-based testing of deep RL agents. It adapts a recurrent state-space model (RSSM) to learn agent behaviour and environment dynamics from per-step training logs, then scores candidate environment configurations by imagined rollouts and a termination-weighted failure score. The authors evaluate on Parking, Humanoid, and DonkeyCar for failure prediction, test generation, and failure diversity.
Why it matters
Instead of predicting pass/fail directly from static configurations, DreamTest learns to model how a test unfolds and estimates failure from imagined episodes. This enables using a world-model surrogate inside existing search-based testing interfaces to prioritize configurations under limited simulator-validation budgets, and to generate behaviourally diverse failures.
Method
- Train an adapted RSSM on per-step DRL training logs: configuration-based initialization, modelling the agent’s actions via behaviour cloning, and predicting termination and failure.
- At inference, for a candidate configuration, generate K parallel imagined rollouts and compute a termination-weighted failure score by aggregating predicted failure at imagined steps.
- Use DreamTest-guided search (ranking, seeded GA, seeded saliency hill climber) under a fixed validation budget and evaluate prediction accuracy and failure diversity via behavioural clustering.
Limitation
When seeds dominate a seeded search (as on DonkeyCar), the authors report no differences between surrogates; when the configuration is the agent’s initial state (as on humanoid), the authors report no difference relative to the MLP within the gradient-based search.
Abstract (from arXiv)
Testing deep reinforcement learning (DRL) agents in cyber-physical systems aims to uncover diverse failures before deployment, but each execution can be expensive. Surrogate-assisted testing reduces this cost by learning to predict which test configurations are likely to fail. Prior surrogates treat the system as a black box and predict pass or fail outcomes directly; we instead model how a test unfolds and estimate failure from an imagined episode. We introduce DreamTest, a world-model surrogate for testing DRL agents. DreamTest adapts a recurrent state-space model to learn agent behaviour and environment dynamics from the agent's training log. Given a candidate configuration, imagined rollouts produce a failure score that guides search without executing every candidate in a simulator or real system. We evaluate DreamTest for failure prediction, test generation, and failure diversity on Parking, Humanoid, and DonkeyCar. Mean area under the precision-recall curve (AUPRC) exceeds the strongest baseline by 97%, 12%, and 39%, respectively, and gains on five out-of-distribution test sets reach 145%, 29%, and 44%. Under the same simulator-validation budget, the best "DreamTest + search" combinations find 29%, 22%, and 79% more novel failures on average. Across clusterings with k = 2-40, failures generated with DreamTest cover the most behavioural clusters for almost all k, indicating that DreamTest consistently discovers behaviourally diverse failures.
Related papers
- World Models Dream of Success: Diagnosing and Repairing Failure Insensitivity in Robot World Models
- Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design
- Mastering Atari with Discrete World Models
- ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation
- Failure Recovery via Belief Updates and World Modeling for Long-Horizon Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC
DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents
Preprint posted: 2026-10-03 · Original record ↗
- arxiv
- 2610.04494
Explicit source versions (1)
- v1 · 2026-10-03
- Preprint available
Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Testing deep reinforcement learning (DRL) agents in cyber-physical systems aims to uncover diverse failures before deployment, but each execution can be expensive. Surrogate-assisted testing reduces this cost by learning to predict which test configurations are likely to fail. Prior surrogates treat the system as a black box and predict pass or fail outcomes directly; we instead model how a test unfolds and estimate failure from an imagined episode. We introduce DreamTest, a world-model surrogate for testing DRL agents. DreamTest adapts a recurrent state-space model to learn agent behaviour and environment dynamics from the agent's training log. Given a candidate configuration, imagined rollouts produce a failure score that guides search without executing every candidate in a simulator or real system. We evaluate DreamTest for failure prediction, test generation, and failure diversity on Parking, Humanoid, and DonkeyCar. Mean area under the precision-recall curve (AUPRC) exceeds the strongest baseline by 97%, 12%, and 39%, respectively, and gains on five out-of-distribution test sets reach 145%, 29%, and 44%. Under the same simulator-validation budget, the best "DreamTest + search" combinations find 29%, 22%, and 79% more novel failures on average. Across clusterings with k = 2-40, failures generated with DreamTest cover the most behavioural clusters for almost all k, indicating that DreamTest consistently discovers behaviourally diverse failures.