Latent Dynamics · 2026-10-05

Mastering Atari 2600 Games with Discovered Options

Erik M. Lintunen, Marlos C. Machado

arXiv:2610.03604PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Wayfarer is a domain-agnostic, online deep RL agent that discovers options from high-dimensional observations using Laplacian representation learning. It couples representation learning, option discovery, and task learning in a single off-policy loop from a single stream of experience. The authors report substantially faster learning and state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games.

Why it matters

The paper addresses a gap in option discovery: general methods that work in large-scale, high-dimensional domains and improve exploration, credit assignment, and generalisation at once, ideally online from single-stream experience. The authors propose a representation-driven option discovery cycle grounded in Laplacian structure, and report strong Atari 2600 results, especially on long-horizon exploration and strategic games.

Method

  • Discover options via Laplacian representation learning from high-dimensional observations: the representation’s smoothest eigenfunctions define intrinsic rewards for option policies.
  • Train three networks jointly online and off-policy from a single replay buffer: a representation network (Laplacian + multi-step inverse-dynamics regularisation), a skill network (intrinsic eigenoptions), and a main network (task-reward values over primitive actions and options).
  • Learn task values with Rainbow-style components and Implicit Quantile Networks (IQN), selecting among primitive actions and options based on estimated return.

Limitation

In Montezuma’s Revenge, the authors observed that Wayfarer can plateau early and suspect implicit step size decay contributes to these plateaus.

Abstract (from arXiv)

Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement learning (RL). However, developing general option discovery methods that are effective in large-scale, high-dimensional domains remains a fundamental challenge. Existing option discovery methods are either confined to relatively simple domains, depend on handcrafted or quasi-symbolic representations, or offer little improvement over learning without options. We present Wayfarer, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control. We show that the resulting options simultaneously improve exploration, accelerate credit assignment, and generalise effectively to unseen settings, enabling substantially faster learning of complex policies. Wayfarer achieves state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games, with the largest gains in games that require long-horizon exploration and strategic behaviour, such as Montezuma's Revenge and Private Eye.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC

preprint

Mastering Atari 2600 Games with Discovered Options

Erik M. Lintunen, Marlos C. Machado

Preprint posted: 2026-10-02 · Original record ↗

arxiv
2610.03604
Explicit source versions (1)
  • v1 · 2026-10-02
  1. Preprint available

    Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement learning (RL). However, developing general option discovery methods that are effective in large-scale, high-dimensional domains remains a fundamental challenge. Existing option discovery methods are either confined to relatively simple domains, depend on handcrafted or quasi-symbolic representations, or offer little improvement over learning without options. We present Wayfarer, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control. We show that the resulting options simultaneously improve exploration, accelerate credit assignment, and generalise effectively to unseen settings, enabling substantially faster learning of complex policies. Wayfarer achieves state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games, with the largest gains in games that require long-horizon exploration and strategic behaviour, such as Montezuma's Revenge and Private Eye.