Latent Dynamics · 2026-11-09

Encoding Memory in World Models with Test-Time Model-Based Reinforcement Learning

Maxime Burchi, Fayaz Ali Dharejo, Radu Timofte

arXiv:2611.09625PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Test-Time Model-Based Reinforcement Learning (TT-MBRL): at test time, they use gradient descent to encode episode-specific memory into world model parameters, learning a distinct world model and policy per evaluation episode. They show improvements on Memory Maze (including a 2D variant) and report new state-of-the-art results on the Memory Maze benchmark, while keeping performance on non-memory tasks like DeepMind Control and Atari 100k.

Why it matters

TT-MBRL targets long-term memory in model-based RL by adapting world model parameters online using episode observations, rather than relying only on contextual representations. This can improve temporal coherence in partially observable settings like Memory Maze and enable episode-specific behavior via a learned world model per episode.

Method

  • Introduce TT-MBRL: learn distinct test-time network parameters (world model, actor, critic) per evaluation episode using gradient descent on episode observations.
  • Use a test-time replay buffer and sample half trajectories from training replay and half from the test-time buffer (regularization); freeze encoder/decoder and drop reconstruction loss at test time.
  • Optimize world model test-time parameters with the prediction objective (omitting reconstruction), then update the actor-critic using imagined rollouts under the adapted world model.

Limitation

The paper states that the effectiveness of TT-MBRL scales proportionally to the number of environment steps collected at test-time.

Abstract (from arXiv)

Model-based reinforcement learning (MBRL) approaches have demonstrated strong performance across diverse domains in recent years, learning accurate world models from agent trajectories. However, despite this progress, current approaches still struggle to surpass human performance on tasks that require long-term memory and planning. In this work, we address the problem of long-term memory in MBRL by moving beyond the commonly studied paradigm of architecture design. We introduce Test-Time Model-Based Reinforcement Learning (TT-MBRL), using gradient descent at test time to encode the memory of episode observations into world model parameters. This enables the agent to learn a distinct world model for each episode, improving memory capabilities compared to traditional approaches that solely rely on contextual memory. We first establish a proof of concept on a simplified two-dimensional variant of the Memory Maze environment, which we refer to as Memory Maze 2D, showing consistent improvements across different world model architectures. We then demonstrate the effectiveness of TT-MBRL in the 3D domain, achieving new state-of-the-art results on the Memory Maze benchmark. Finally, we show that TT-MBRL maintains strong performance on non-memory benchmarks such as DeepMind Control and Atari 100k.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-06 · Indexed by Pub-AI: 2026-11-09 UTC

preprint

Encoding Memory in World Models with Test-Time Model-Based Reinforcement Learning

Maxime Burchi, Fayaz Ali Dharejo, Radu Timofte

Preprint posted: 2026-11-06 · Original record ↗

arxiv
2611.09625
Explicit source versions (1)
  • v1 · 2026-11-06
  1. Preprint available

    Source date: 2026-11-06 · First detected: 2026-11-09 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Model-based reinforcement learning (MBRL) approaches have demonstrated strong performance across diverse domains in recent years, learning accurate world models from agent trajectories. However, despite this progress, current approaches still struggle to surpass human performance on tasks that require long-term memory and planning. In this work, we address the problem of long-term memory in MBRL by moving beyond the commonly studied paradigm of architecture design. We introduce Test-Time Model-Based Reinforcement Learning (TT-MBRL), using gradient descent at test time to encode the memory of episode observations into world model parameters. This enables the agent to learn a distinct world model for each episode, improving memory capabilities compared to traditional approaches that solely rely on contextual memory. We first establish a proof of concept on a simplified two-dimensional variant of the Memory Maze environment, which we refer to as Memory Maze 2D, showing consistent improvements across different world model architectures. We then demonstrate the effectiveness of TT-MBRL in the 3D domain, achieving new state-of-the-art results on the Memory Maze benchmark. Finally, we show that TT-MBRL maintains strong performance on non-memory benchmarks such as DeepMind Control and Atari 100k.