Latent Dynamics · 2026-11-02

Before We Adapt: The Zero-Shot Generalization Bottleneck in Continual World Models

Amir Mesbah, Di Wu

arXiv:2610.41041PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study zero-shot reuse of learned world models in continual RL when behavioral context is fixed but physical context changes. Using Dyna-Q and TD-MPC on controlled FourRooms tasks, they find that reuse can degrade substantially under physical-context changes, and that this depends strongly on the observation/representation. Ablations and latent-space diagnostics suggest learned representations do not automatically provide invariances needed for reuse.

Why it matters

Continual model-based RL benefits from reusing previously learned dynamics when environments change. This paper argues that such reuse requires zero-shot generalization, and shows experimentally that representation sensitivity to physical context can create a bottleneck even when behavioral rules are unchanged. The results motivate studying context-invariant or factorized representations and better diagnostics for when reuse is no longer reliable.

Method

  • Formulate continual world-model learning as a contextual MDP, decomposing context into behavioral (rules) and physical (geometry/layout/spatial instantiation) components, and focus on the reuse regime where behavioral context is fixed.
  • Train Dyna-Q and TD-MPC in one FourRooms context and evaluate zero-shot in other contexts without parameter updates, reporting a log generalization gap from one-step dynamics prediction error.
  • Run ablations separating translation-only, layout-only, and combined physical changes, plus TD-MPC latent-space diagnostics (linear probes and latent alignment) to assess representation invariance across contexts.

Limitation

The paper focuses experimentally on the reuse regime with fixed behavioral context; it states that extensions to changes in behavioral context remain to be studied and may require adaptation beyond zero-shot reuse.

Abstract (from arXiv)

Continual reinforcement learning (CRL) requires agents to operate under environmental changes while reusing prior knowledge whenever possible. We argue that model-based reinforcement learning provides a promising framework for this setting because learned world models (WMs) can, in principle, support change detection, knowledge reuse, adaptation, and acquisition of new knowledge. However, such benefits depend heavily on whether the learned model can generalize beyond the contexts in which it was trained. In this work, we propose to formulate this problem with a contextual Markov decision process in which context is decomposed into behavioral and physical components. This view organizes continual WM learning into reuse, adaptation, and acquisition of new knowledge. As a first step, we study direct reuse when behavioral rules remain unchanged. Using Dyna-Q and TD-MPC in controlled FourRooms tasks, we find that zero-shot reuse can degrade substantially under physical-context changes and that this behavior depends strongly on the observation representation used by the WM. Additional ablation studies and latent-space diagnostics suggest that learned representations do not automatically provide the invariances needed for reuse. These results identify representation generalization as an important bottleneck for reliable world-model reuse in CRL.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-02 · Indexed by Pub-AI: 2026-11-02 UTC

preprint

Before We Adapt: The Zero-Shot Generalization Bottleneck in Continual World Models

Amir Mesbah, Di Wu

First announced by arXiv: 2026-11-02 · Original record ↗

arxiv
2610.41041
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-02 · First detected: 2026-11-02 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2610.41041v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Continual reinforcement learning (CRL) requires agents to operate under environmental changes while reusing prior knowledge whenever possible. We argue that model-based reinforcement learning provides a promising framework for this setting because learned world models (WMs) can, in principle, support change detection, knowledge reuse, adaptation, and acquisition of new knowledge. However, such benefits depend heavily on whether the learned model can generalize beyond the contexts in which it was trained. In this work, we propose to formulate this problem with a contextual Markov decision process in which context is decomposed into behavioral and physical components. This view organizes continual WM learning into reuse, adaptation, and acquisition of new knowledge. As a first step, we study direct reuse when behavioral rules remain unchanged. Using Dyna-Q and TD-MPC in controlled FourRooms tasks, we find that zero-shot reuse can degrade substantially under physical-context changes and that this behavior depends strongly on the observation representation used by the WM. Additional ablation studies and latent-space diagnostics suggest that learned representations do not automatically provide the invariances needed for reuse. These results identify representation generalization as an important bottleneck for reliable world-model reuse in CRL.