Latent Dynamics · 2026-10-05
Mastering Atari 2600 Games with Discovered Options
Erik M. Lintunen, Marlos C. Machado
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Wayfarer is a domain-agnostic, online deep RL agent that discovers options from high-dimensional observations using Laplacian representation learning. It couples representation learning, option discovery, and task learning in a single off-policy loop from a single stream of experience. The authors report substantially faster learning and state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games.
Why it matters
The paper addresses a gap in option discovery: general methods that work in large-scale, high-dimensional domains and improve exploration, credit assignment, and generalisation at once, ideally online from single-stream experience. The authors propose a representation-driven option discovery cycle grounded in Laplacian structure, and report strong Atari 2600 results, especially on long-horizon exploration and strategic games.
Method
- Discover options via Laplacian representation learning from high-dimensional observations: the representation’s smoothest eigenfunctions define intrinsic rewards for option policies.
- Train three networks jointly online and off-policy from a single replay buffer: a representation network (Laplacian + multi-step inverse-dynamics regularisation), a skill network (intrinsic eigenoptions), and a main network (task-reward values over primitive actions and options).
- Learn task values with Rainbow-style components and Implicit Quantile Networks (IQN), selecting among primitive actions and options based on estimated return.
Limitation
In Montezuma’s Revenge, the authors observed that Wayfarer can plateau early and suspect implicit step size decay contributes to these plateaus.
Abstract (from arXiv)
Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement learning (RL). However, developing general option discovery methods that are effective in large-scale, high-dimensional domains remains a fundamental challenge. Existing option discovery methods are either confined to relatively simple domains, depend on handcrafted or quasi-symbolic representations, or offer little improvement over learning without options. We present Wayfarer, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control. We show that the resulting options simultaneously improve exploration, accelerate credit assignment, and generalise effectively to unseen settings, enabling substantially faster learning of complex policies. Wayfarer achieves state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games, with the largest gains in games that require long-horizon exploration and strategic behaviour, such as Montezuma's Revenge and Private Eye.
Related papers
- Model-Based Reinforcement Learning for Atari
- Reinforcement Learning for the Game of Balatro: A Fast Simulator and Two-Policy Training Approach
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
- JAXtari: High-Throughput and Easy-to-Modify Arcade Learning Environment
- Mastering Atari with Discrete World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
Mastering Atari 2600 Games with Discovered Options
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.03604
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement learning (RL). However, developing general option discovery methods that are effective in large-scale, high-dimensional domains remains a fundamental challenge. Existing option discovery methods are either confined to relatively simple domains, depend on handcrafted or quasi-symbolic representations, or offer little improvement over learning without options. We present Wayfarer, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control. We show that the resulting options simultaneously improve exploration, accelerate credit assignment, and generalise effectively to unseen settings, enabling substantially faster learning of complex policies. Wayfarer achieves state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games, with the largest gains in games that require long-horizon exploration and strategic behaviour, such as Montezuma's Revenge and Private Eye.
