Driving · 2026-10-06

DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation

Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, Stefan Wermter

arXiv:2610.04540PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DreamFormer is a model-based, language-conditioned robotic manipulation agent that learns a task-agnostic Transformer world model from unstructured play data, then trains a policy inside the world model’s latent imagination using Dream Imitation with an intrinsic reward aligning imagined rollouts to expert demonstrations. It encodes full-resolution multi-view observations into a single token per timestep via a scaled categorical VAE. On CALVIN, it outperforms LUMOS and nearly doubles HULC on zero-shot transfer.

Why it matters

The authors aim to improve transfer in language-conditioned robotic manipulation by decoupling dynamics learning (task-agnostic, reward-free world model) from policy learning (imagination-based Dream Imitation with an intrinsic reward). They also address the long-horizon cost of Transformer world models by tokenizing high-resolution multi-view observations into a single token per timestep, keeping imagination rollout costs bounded by horizon length.

Method

  • Train a task-agnostic GPT-style Transformer world model on unstructured play data using a scaled categorical VAE (ResNet-18 encoder; 64 categories × 32 classes) and reconstruction + entropy-regularized latent dynamics learning.
  • Learn a language-conditioned multi-task policy entirely inside imagination by optimizing an intrinsic reward that aligns imagined policy rollouts with expert latent trajectories (Dream Imitation).
  • For long-horizon tractability, encode each multi-view robotic observation into one Transformer token per timestep, avoiding both spatial downsampling and multi-token representations used by prior Transformer world models.

Limitation

The authors state: “Our evaluation is confined to a simulated benchmark, and whether the same representation transfers to real robots is untested.”

Abstract (from arXiv)

We introduce DreamFormer, a model-based agent that acquires language-conditioned, multi-task skills by imitating expert demonstrations within the latent imagination of a learned world model. DreamFormer first learns a task-agnostic Transformer world model from unstructured play data, then acquires task-specific behaviors by optimizing an intrinsic reward that aligns agent-generated rollouts with expert demonstrations in latent space. Since the policy is trained on-policy inside imagination, it is exposed to its own errors during training, mitigating the covariate shift inherent to offline behavioral cloning. To make long-horizon imagination affordable, DreamFormer encodes a high-resolution multi-view robotic observation into a single input token, avoiding both spatial downsampling and the multi-token representations used by prior Transformer world models. On the long-horizon CALVIN benchmark, DreamFormer outperforms LUMOS, the comparable model-based agent, on single-environment evaluation (2.52 vs 2.34 average tasks completed per chain of five) while keeping imagination rollouts tractable. Against HULC, the behavior cloning baseline, it nearly doubles performance on zero-shot transfer to an unseen environment (1.30 vs 0.67), indicating that dynamics learned by the world model transfer more readily than a directly cloned policy. This is consistent with the role attributed to internal models in biological agents, where a model of environment dynamics supports behavior in situations not previously encountered.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC

preprint

DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation

Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, Stefan Wermter

Preprint posted: 2026-10-03 · Original record ↗

arxiv
2610.04540
Explicit source versions (1)
  • v1 · 2026-10-03
  1. Preprint available

    Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

We introduce DreamFormer, a model-based agent that acquires language-conditioned, multi-task skills by imitating expert demonstrations within the latent imagination of a learned world model. DreamFormer first learns a task-agnostic Transformer world model from unstructured play data, then acquires task-specific behaviors by optimizing an intrinsic reward that aligns agent-generated rollouts with expert demonstrations in latent space. Since the policy is trained on-policy inside imagination, it is exposed to its own errors during training, mitigating the covariate shift inherent to offline behavioral cloning. To make long-horizon imagination affordable, DreamFormer encodes a high-resolution multi-view robotic observation into a single input token, avoiding both spatial downsampling and the multi-token representations used by prior Transformer world models. On the long-horizon CALVIN benchmark, DreamFormer outperforms LUMOS, the comparable model-based agent, on single-environment evaluation (2.52 vs 2.34 average tasks completed per chain of five) while keeping imagination rollouts tractable. Against HULC, the behavior cloning baseline, it nearly doubles performance on zero-shot transfer to an unseen environment (1.30 vs 0.67), indicating that dynamics learned by the world model transfer more readily than a directly cloned policy. This is consistent with the role attributed to internal models in biological agents, where a model of environment dynamics supports behavior in situations not previously encountered.