Robotics · 2026-10-12

Dream to Remember: Continual Steering of Generalist Robot Policies with World Models

Bahey Tharwat, Iman Nematollahi, Marvin Alles, Xingyuan Zhang, Akshay L Chandra, Abhinav Valada

arXiv:2610.13816PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Dream to Remember (D2R) continually steers a frozen generalist robot policy toward sequential new tasks while retaining earlier skills. It learns a small steering module using imagined rollouts from a frozen world model; the same world model rehearses prior tasks by regenerating their rollouts on demand, regularizing the steering module via a KL penalty. On CALVIN it improves 8 tasks entirely in imagination, and on LIBERO-90 it learns unseen tasks in the environment while preserving the policy’s earlier skills.

Why it matters

The authors address catastrophic forgetting in continual adaptation of generalist robot policies trained with imitation data. D2R avoids replay buffers and task-specific retraining by using a frozen world model as “memory” to rehearse prior tasks through regeneration. The paper also studies how rehearsal can work even when the world model is too inaccurate for policy optimization, enabling continual steering with off-the-shelf policies and world models.

Method

  • Continually train a small input-noise steering module π^W(w|s,ℓ) on top of a frozen generative policy π^W_dp, optimizing for current-task return in imagination when world-model fidelity allows.
  • Prevent forgetting by rehearsing prior tasks in a frozen world model: generate imagined states for retention tasks and regularize the steering module using a KL penalty to match the previous steering behavior (plus default anchoring to N(0,I) for never-steered tasks).
  • Choose the world model role based on fidelity: if faithful enough, learn the new task entirely in imagination; otherwise learn in the environment and use the world model only for rehearsal.

Limitation

The authors report that on LIBERO-90, A2World is not accurate enough for policy optimization, so D2R learns new tasks in the environment there (using the world model only for rehearsal).

Abstract (from arXiv)

Generalist robot policies are trained with imitation learning and are bounded by the quality and coverage of their demonstrations. Reinforcement learning can improve and extend them, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy. When tasks arrive sequentially, both approaches suffer from catastrophic forgetting. Fine-tuning on a task degrades prior tasks, and steering only shifts forgetting to the shared steering module. We propose Dream to Remember (D2R), which continually steers a frozen generalist policy toward new tasks, while retaining its earlier skills. As the steering module learns a new task, a frozen pretrained world model rehearses prior tasks through imagined rollouts, anchoring the module to its earlier behavior without a replay buffer or new data. When the world model is faithful enough, D2R learns the new task entirely in imagination. Otherwise, it learns the task in the environment while using the world model only to rehearse prior tasks. On CALVIN, D2R improves eight tasks in sequence entirely in imagination, raising average success from 63.7% to 74.8%, while LoRA fine-tuning on the same sequence drops it to 11.7%. With a frozen off-the-shelf generalist policy and world model, $\pi_0$ and A2World, D2R learns unseen LIBERO-90 tasks in the environment while keeping $\pi_0$'s performance on the LIBERO suites it was fine-tuned on, even though the world model is not accurate enough for policy optimization. Finally, on a real Franka robot, D2R improves three tasks in sequence entirely in imagination while retaining the earlier improvements.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC

preprint

Dream to Remember: Continual Steering of Generalist Robot Policies with World Models

Bahey Tharwat, Iman Nematollahi, Marvin Alles, Xingyuan Zhang, Akshay L Chandra, Abhinav Valada

Preprint posted: 2026-10-09 · Original record ↗

arxiv
2610.13816
Explicit source versions (1)
  • v1 · 2026-10-09
  1. Preprint available

    Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Generalist robot policies are trained with imitation learning and are bounded by the quality and coverage of their demonstrations. Reinforcement learning can improve and extend them, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy. When tasks arrive sequentially, both approaches suffer from catastrophic forgetting. Fine-tuning on a task degrades prior tasks, and steering only shifts forgetting to the shared steering module. We propose Dream to Remember (D2R), which continually steers a frozen generalist policy toward new tasks, while retaining its earlier skills. As the steering module learns a new task, a frozen pretrained world model rehearses prior tasks through imagined rollouts, anchoring the module to its earlier behavior without a replay buffer or new data. When the world model is faithful enough, D2R learns the new task entirely in imagination. Otherwise, it learns the task in the environment while using the world model only to rehearse prior tasks. On CALVIN, D2R improves eight tasks in sequence entirely in imagination, raising average success from 63.7% to 74.8%, while LoRA fine-tuning on the same sequence drops it to 11.7%. With a frozen off-the-shelf generalist policy and world model, $π_0$ and A2World, D2R learns unseen LIBERO-90 tasks in the environment while keeping $π_0$'s performance on the LIBERO suites it was fine-tuned on, even though the world model is not accurate enough for policy optimization. Finally, on a real Franka robot, D2R improves three tasks in sequence entirely in imagination while retaining the earlier improvements.