Latent Dynamics · 2026-10-22

Second Thoughts: Exploring and Planning through Comparative Imagination in World Models

Hieu Tran, Hong Yu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Second Thoughts (SDI) links exploration and planning decisions in world-model RL by comparing imagined action “siblings” to repeated rollouts of the actor’s fixed sequence from the same latent state. The spread of sibling returns drives an exploration bonus, and a gate accepts a planned action only when its predicted advantage exceeds actor-repeat variability by a fixed factor. Evaluation uses the learned actor alone.

Why it matters

SDI provides a unified decision rule for (1) when different imagined actions are informative enough to affect training via an exploration bonus and (2) when a searched plan should replace the actor during data collection. It avoids an additional disagreement ensemble and keeps evaluation actor-only, while reducing return degradation in paired-state tests.

Method

Actor-referenced imagined returns: siblings are candidate action sequences imagined from the same latent state; actor-repeat rollouts measure stochastic return variation under fixed actor behavior.
Unified exploration + plan acceptance: exploration bonus scales with sibling return spread; a planned action is executed only if its predicted advantage exceeds actor-reference variability by factor κ.
No extra disagreement ensemble: SDI reuses the base world model, actor, and critic; exploration and gating use the actor-based reference, while evaluation uses the actor alone.

Limitation

Repeated actor rollouts quantify stochastic return variation for one fixed imagined sequence; they do not capture model bias shared by all rollouts. The acceptance threshold κ is an empirical operating point rather than a statistical confidence level.

Abstract (from arXiv)

World models let agents compare imagined outcomes before acting, but an agent must still decide which alternatives to explore and when a proposed plan should replace its learned policy. We introduce SDI (Sibling Disagreement in Imagination), which connects these decisions through a shared actor-based reference. SDI compares action sequences imagined from the same model state, using repeated rollouts of the actor's own sequence to measure variation under fixed behavior. This reference scales both the spread across candidate returns, which provides an exploration signal, and a proposed plan's predicted advantage, which governs action acceptance. The agent executes a planned action when its advantage exceeds the reference variability by a fixed factor; otherwise, it follows the actor. SDI reuses the base agent's world model, actor, and critic without requiring an additional disagreement ensemble. Exploration and planning guide training data collection, while evaluation uses the learned actor alone. Across Meta-World, Adroit, and DM-Control, SDI achieves a higher observed task-average score than every compared baseline in each suite and matches or exceeds the strongest baseline's performance on 15 of 18 tasks. Component ablations and robustness analyses further examine the roles of exploration, planning exposure, and action acceptance. Paired evaluations from identical simulator states show that the gate improves both recovery from actor failures and preservation of successful behavior relative to ungated planning, while substantially reducing return degradation. Together, these results support comparing imagined alternatives against a shared actor reference as a unified approach to exploring and planning with world models.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-22 UTC

preprint