Driving · 2026-10-01
Reinforcement Learning with Complex (valued) Memories
Sathya Kamesh Bhethanabhotla, Efstratios Gavves, Andr\'e Biedenkapp
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors revisit unitary recurrent networks (uRNNs), using complex-valued, norm-preserving unitary dynamics as a drop-in recurrent cell for recurrent PPO. They propose three uRNN variants and show significant gains over GRU-PPO and PPO-LD on partially observable memory-improvable tasks, with reported up to 2–3× reward on RockSample and Craftax. Code is released at github.com/Sathya98/qurl.
Why it matters
For partially observable RL, memory and stable long-sequence information flow are central challenges. The paper tests whether complex-valued unitary recurrence—promoting stable norm-preserving dynamics and enabling richer representations via phase—can improve recurrent PPO without changing encoders/heads/PPO objective. It also explores a phase-aware policy head inspired by quantum measurement (Born rule).
Method
- Replace the GRU cell in recurrent PPO (from POBAX) with a uRNN carrying a complex hidden state and unitary transitions, leaving the rest of the agent unchanged.
- Evaluate three uRNN variants: uRNN (original unitary parameterization with constant U), EUNN (tunable unitary recurrence), and ICuRNN (input-conditioned unitary transformation U(x_t)).
- Provide a phase-aware policy interpretation for complex hidden states via a Born-rule-like sampling method (quantum measurement parallel).
Abstract (from arXiv)
Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from observations and maintain a memory to make effective decisions. While there exist many approaches ranging from gated recurrence to attention mechanisms and model-based RL, the search for effective representational techniques that can capture long-term dependencies remains an active area of research. In this work we revisit Unitary recurrent networks (uRNNs) [Arjovsky et al., 2016, Jing et al., 2017], that demonstrated superior gradient flow and associative recall, expressing the recurrence and the hidden state in a complex vector space. Their norm preserving unitary dynamics enable information propagation through long sequences. To this end, we propose three different versions of uRNNs as drop-in replacements for recurrent PPO architectures, and demonstrate that the simple recurrence and the added degree of freedom from the phase of the complex representations enable significant gains over baselines on several memory-improvable tasks, including continuous control. We further explore how to preserve the phase information of the complex hidden state for a phase-aware policy by drawing a parallel to how quantum states are measured. With our methods reaching up to 2-3 $\times$ the reward in environments like rocksample and Craftax compared to the baselines, this work points towards an exciting new direction of representations for RL and the problem of partial observability. Code is available at: https://github.com/Sathya98/qurl
Related papers
- Self-Confirming Superposition Traps in Reinforcement Learning
- Learning Transferable Policies from Action-free Time Series Through Dynamical Embeddings
- Linear Recurrent Memory Suffices to Distil a World-Model Policy for Robot Air Hockey
- Looped Reasoning Finishes Earlier Than You Think
- Dynamics-Aligned Representation Learning under Induced Partial Observability
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-29 · Indexed by Pub-AI: 2026-10-01 UTC
Reinforcement Learning with Complex (valued) Memories
Preprint posted: 2026-09-29 · Original record ↗
- arxiv
- 2609.38598
Explicit source versions (1)
- v1 · 2026-09-29
- Preprint available
Source date: 2026-09-29 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from observations and maintain a memory to make effective decisions. While there exist many approaches ranging from gated recurrence to attention mechanisms and model-based RL, the search for effective representational techniques that can capture long-term dependencies remains an active area of research. In this work we revisit Unitary recurrent networks (uRNNs) [Arjovsky et al., 2016, Jing et al., 2017], that demonstrated superior gradient flow and associative recall, expressing the recurrence and the hidden state in a complex vector space. Their norm preserving unitary dynamics enable information propagation through long sequences. To this end, we propose three different versions of uRNNs as drop-in replacements for recurrent PPO architectures, and demonstrate that the simple recurrence and the added degree of freedom from the phase of the complex representations enable significant gains over baselines on several memory-improvable tasks, including continuous control. We further explore how to preserve the phase information of the complex hidden state for a phase-aware policy by drawing a parallel to how quantum states are measured. With our methods reaching up to 2-3 $\times$ the reward in environments like rocksample and Craftax compared to the baselines, this work points towards an exciting new direction of representations for RL and the problem of partial observability. Code is available at: https://github.com/Sathya98/qurl