Latent Dynamics · 2026-11-11

Building World Models from Intrinsic Motivation

Reece Keller, Declan Kutscher, Xaq Pitkow, Aran Nayebi

arXiv:2611.13113PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors integrate 24 intrinsic objectives (curiosity, entropy, competency) into a common DreamerV3 model-based RL framework, alongside three representation-learning strategies (observation reconstruction, next-embedding prediction, current-embedding redundancy reduction). They evaluate zero-shot, few-shot, and multi-task control across locomotion, manipulation, and open-ended gaming. Curiosity with observation reconstruction gives the strongest overall family profile; episodic+lifelong novelty yields the best compositional vision transfer and adaptation.

Why it matters

It clarifies how different intrinsic motivations interact with world-model representation learning to support downstream control. The study provides a unified comparison across many intrinsic objectives, evaluated via control (zero-shot, transfer, and adaptation) rather than only proxy measures like coverage, and motivates jointly designing intrinsic objectives and representation learning.

Method

  • Integrate 24 intrinsic objectives into a common model-based DreamerV3 framework using an Intrinsic Drive Module (IDM) that scores imagined transitions while using the deterministic RSSM component for intrinsic features.
  • Compare three world-model representation anchors: DreamerV3 observation reconstruction, NE-Dreamer next-embedding prediction, and R2-Dreamer current-embedding redundancy reduction (with redundancy reduction to prevent collapse).
  • Evaluate the resulting world models via zero-shot control, multi-task transfer, and few-shot adaptation across DeepMind Control continuous-control tasks (locomotion/manipulation) and Craftax open-world gaming; for adaptation, retain the pretrained world model and replay but reinitialize task heads.

Limitation

They report a scaling mismatch: “more policy training in a pretrained world model impairs subsequent adaptation in Craftax,” which they attribute to a weakness in Dreamer’s ability to maintain a strong persistent memory in abstract, compositional tasks.

Abstract (from arXiv)

In the absence of extrinsic rewards or task supervision, human behavior is guided by intrinsic motivation---neural signals that drive curiosity, creativity, play, and other essential facets of intelligence. Intrinsic motivation plays an important developmental role in building robust world models that enable generalization and adaptation. Yet existing studies leave open how a broad range of intrinsic motivations interacts with world-model representation learning to support downstream control. This limits our understanding of which intrinsically collected experiences build reusable models. To address these gaps, we integrate 24 intrinsic objectives, including established methods and explicit variants spanning curiosity, entropy, and competency, into a common model-based reinforcement learning framework. We evaluate the world models they build through zero-shot, few-shot, and multi-task control in locomotion, manipulation, and open-ended gaming environments, using three representation-learning strategies: observation reconstruction, latent prediction, and redundancy reduction. Curiosity-driven exploration with observation reconstruction yields the strongest aggregate family profile, although the best individual pairing depends on the experiment. In compositional vision-based tasks, intrinsic motivation derived from both episodic and lifelong novelty---providing a complementary memory to the world model's generic recurrent state---achieves the best zero-shot transfer and few-shot adaptation. These findings motivate designing intrinsic motivation and representation learning jointly. Our results demonstrate world models as a powerful tool for measuring the quality of intrinsically motivated experience, and positions intrinsic motivation as a key ingredient for building world models autonomously.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-11 · Indexed by Pub-AI: 2026-11-11 UTC

preprint

Building World Models from Intrinsic Motivation

Reece Keller, Declan Kutscher, Xaq Pitkow, Aran Nayebi

First announced by arXiv: 2026-11-11 · Original record ↗

arxiv
2611.13113
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-11 · First detected: 2026-11-11 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.13113v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

In the absence of extrinsic rewards or task supervision, human behavior is guided by intrinsic motivation---neural signals that drive curiosity, creativity, play, and other essential facets of intelligence. Intrinsic motivation plays an important developmental role in building robust world models that enable generalization and adaptation. Yet existing studies leave open how a broad range of intrinsic motivations interacts with world-model representation learning to support downstream control. This limits our understanding of which intrinsically collected experiences build reusable models. To address these gaps, we integrate 24 intrinsic objectives, including established methods and explicit variants spanning curiosity, entropy, and competency, into a common model-based reinforcement learning framework. We evaluate the world models they build through zero-shot, few-shot, and multi-task control in locomotion, manipulation, and open-ended gaming environments, using three representation-learning strategies: observation reconstruction, latent prediction, and redundancy reduction. Curiosity-driven exploration with observation reconstruction yields the strongest aggregate family profile, although the best individual pairing depends on the experiment. In compositional vision-based tasks, intrinsic motivation derived from both episodic and lifelong novelty---providing a complementary memory to the world model's generic recurrent state---achieves the best zero-shot transfer and few-shot adaptation. These findings motivate designing intrinsic motivation and representation learning jointly. Our results demonstrate world models as a powerful tool for measuring the quality of intrinsically motivated experience, and positions intrinsic motivation as a key ingredient for building world models autonomously.