Latent Dynamics · 2020-10-05
Mastering Atari with Discrete World Models
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba
arXiv:2010.02193PDFProject page
Auto-summarized: this summary was generated by a language model from the paper’s text and has not been reviewed by an editor. Check the paper before relying on it. Benchmark numbers appear only after a human has verified them.
TL;DR
DreamerV2 learns behaviors purely from predictions in the latent space of a world model that uses discrete latent representations and is trained separately from the policy, and is the first such agent to reach human-level performance on the 55-game Atari benchmark.
Why it matters
Closes a long-standing gap: earlier attempts at learning accurate Atari world models did not reach competitive performance. The authors report it surpasses the top single-GPU model-free agents IQN and Rainbow at the same compute budget.
Method
- An evolution of Dreamer with a world model that uses discrete latent representations.
- Balancing terms inside the KL loss, among other small modifications to Dreamer.
- Policy is learned entirely from imagined latent trajectories.
Lineage
- builds on → Dream to Control: Learning Behaviors by Latent Imagination (An evolution of the Dreamer agent (called DreamerV1 in the paper).)
- ← builds on by Mastering Diverse Domains through World Models (Third generation of Dreamer, following DreamerV2.)
Abstract (from arXiv)
Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance of the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pixel inputs.