World Models

TL;DR

Trains a VAE to compress each frame and a recurrent mixture-density network to predict the next latent, then fits a tiny linear controller on their features; the agent can also be trained entirely inside the model's own generated 'dream' and transferred back to the real environment.

Why it matters

A simple, modular recipe (vision, memory, controller) for learning a compact world model from pixels, and an early demonstration that a policy trained purely in imagination can transfer to the real environment. Later papers in this list, such as PlaNet, cite it.

Method

• Vision (V): a variational autoencoder compresses each frame into a latent vector z.
• Memory (M): an MDN-RNN predicts the next latent given the current latent, action and hidden state.
• Controller (C): a small linear layer over [z, h] is optimized with the evolution strategy CMA-ES.

Limitation

The authors note the LSTM-based world model has limited capacity and may not store all recorded information, and that the VAE may not capture task-relevant features (it failed to reproduce task-relevant tiles on the road in CarRacing).

Lineage

• ← builds on by Learning Latent Dynamics for Planning from Pixels (Cites World Models (Ha & Schmidhuber, 2018) and uses its convolutional/deconvolutional networks.)

Related papers

• Dream to Control: Learning Behaviors by Latent Imagination
• Diffusion for World Modeling: Visual Details Matter in Atari
• DayDreamer: World Models for Physical Robot Learning
• Mastering Atari with Discrete World Models
• Learning Interactive Real-World Simulators