Driving · 2026-10-06
DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation
Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, Stefan Wermter
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DreamFormer is a model-based, language-conditioned robotic manipulation agent that learns a task-agnostic Transformer world model from unstructured play data, then trains a policy inside the world model’s latent imagination using Dream Imitation with an intrinsic reward aligning imagined rollouts to expert demonstrations. It encodes full-resolution multi-view observations into a single token per timestep via a scaled categorical VAE. On CALVIN, it outperforms LUMOS and nearly doubles HULC on zero-shot transfer.
Why it matters
The authors aim to improve transfer in language-conditioned robotic manipulation by decoupling dynamics learning (task-agnostic, reward-free world model) from policy learning (imagination-based Dream Imitation with an intrinsic reward). They also address the long-horizon cost of Transformer world models by tokenizing high-resolution multi-view observations into a single token per timestep, keeping imagination rollout costs bounded by horizon length.
Method
- Train a task-agnostic GPT-style Transformer world model on unstructured play data using a scaled categorical VAE (ResNet-18 encoder; 64 categories × 32 classes) and reconstruction + entropy-regularized latent dynamics learning.
- Learn a language-conditioned multi-task policy entirely inside imagination by optimizing an intrinsic reward that aligns imagined policy rollouts with expert latent trajectories (Dream Imitation).
- For long-horizon tractability, encode each multi-view robotic observation into one Transformer token per timestep, avoiding both spatial downsampling and multi-token representations used by prior Transformer world models.
Limitation
The authors state: “Our evaluation is confined to a simulated benchmark, and whether the same representation transfers to real robots is untested.”
Abstract (from arXiv)
We introduce DreamFormer, a model-based agent that acquires language-conditioned, multi-task skills by imitating expert demonstrations within the latent imagination of a learned world model. DreamFormer first learns a task-agnostic Transformer world model from unstructured play data, then acquires task-specific behaviors by optimizing an intrinsic reward that aligns agent-generated rollouts with expert demonstrations in latent space. Since the policy is trained on-policy inside imagination, it is exposed to its own errors during training, mitigating the covariate shift inherent to offline behavioral cloning. To make long-horizon imagination affordable, DreamFormer encodes a high-resolution multi-view robotic observation into a single input token, avoiding both spatial downsampling and the multi-token representations used by prior Transformer world models. On the long-horizon CALVIN benchmark, DreamFormer outperforms LUMOS, the comparable model-based agent, on single-environment evaluation (2.52 vs 2.34 average tasks completed per chain of five) while keeping imagination rollouts tractable. Against HULC, the behavior cloning baseline, it nearly doubles performance on zero-shot transfer to an unseen environment (1.30 vs 0.67), indicating that dynamics learned by the world model transfer more readily than a directly cloned policy. This is consistent with the role attributed to internal models in biological agents, where a model of environment dynamics supports behavior in situations not previously encountered.
Related papers
- Dream to Control: Learning Behaviors by Latent Imagination
- The Planning Limits of Latent World Models
- DayDreamer: World Models for Physical Robot Learning
- Dream to Remember: Continual Steering of Generalist Robot Policies with World Models
- DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC
DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation
Preprint posted: 2026-10-03 · Original record ↗
- arxiv
- 2610.04540
Explicit source versions (1)
- v1 · 2026-10-03
- Preprint available
Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We introduce DreamFormer, a model-based agent that acquires language-conditioned, multi-task skills by imitating expert demonstrations within the latent imagination of a learned world model. DreamFormer first learns a task-agnostic Transformer world model from unstructured play data, then acquires task-specific behaviors by optimizing an intrinsic reward that aligns agent-generated rollouts with expert demonstrations in latent space. Since the policy is trained on-policy inside imagination, it is exposed to its own errors during training, mitigating the covariate shift inherent to offline behavioral cloning. To make long-horizon imagination affordable, DreamFormer encodes a high-resolution multi-view robotic observation into a single input token, avoiding both spatial downsampling and the multi-token representations used by prior Transformer world models. On the long-horizon CALVIN benchmark, DreamFormer outperforms LUMOS, the comparable model-based agent, on single-environment evaluation (2.52 vs 2.34 average tasks completed per chain of five) while keeping imagination rollouts tractable. Against HULC, the behavior cloning baseline, it nearly doubles performance on zero-shot transfer to an unseen environment (1.30 vs 0.67), indicating that dynamics learned by the world model transfer more readily than a directly cloned policy. This is consistent with the role attributed to internal models in biological agents, where a model of environment dynamics supports behavior in situations not previously encountered.
