Driving · 2026-10-30
Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information
Daniel Strauss, Nikolaos Sotirakis, Zhenshan Bing, Alois Knoll
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose “value of information” for credit assignment and retrieval in memory-based RL. They estimate information value from archived rollouts using a learned world model. When read at the observation, it yields a provable lower bound that floors TD targets and attributes the floor to the observation that earned it; read at the decision, it defines an optimal retrieval set. A pilot “Memory Navigation” system uses a reduced-context retrieval setup.
Why it matters
The paper targets two RL memory problems: expensive attention over large stores and observation credit assignment where credit must cross long sequences. By grounding both retrieval and temporal credit in one quantity (information value) and deriving a provable TD-target floor, the approach aims to train retrieval using signals tied to which observation information mattered and when.
Method
- Define information value as the change in achievable return when an information set (history items) is made available at a decision, and relate it to optimal action values and Bellman backups in a POMDP.
- Learn a world model from archived rollouts, then estimate information value by evaluating expected return under masked knowledge states and regression against Bellman targets.
- For acquisition credit, estimate a lower-bound TD target that combines a baseline floor with a credit term derived from the observation’s information value; for retrieval, select the optimal retrieval set via an information-value objective penalized by retrieval costs.
Limitation
Neither the information-value estimator nor the floored target is implemented yet, so the expected faster convergence from using the update is a hypothesis. Bounds are for exact quantities; with learned models and noisy estimates, maximizing can exploit errors and make the estimated floor biased upward. Additional practical assumptions include that information value is defined under the exploration policy, requiring storing policy versions in the history if the archive’s policy changes.
Abstract (from arXiv)
Memory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence. We argue that both are governed by one quantity, the value of information, and propose to estimate it from archived rollouts with a learned world model. We obtain that, when read at the observation, it yields a provable lower bound on the optimal value of the acquiring action, which floors its TD target and attributes this floor to the observation that earned it. Read at the decision, it defines the optimal retrieval set. A pilot retrieval system, to be trained and improved with this signal, solves a memory task with reduced context.
Related papers
- MemoWM: How World Models Change What Agents Need to Remember
- T$^2$Mem: Learning Test-Time Memory for Robotics
- Considering Context: When World Models Need Context Encoders
- Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies
- Heavy-Tailed Memory Traces in Long-Horizon Language Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-10-30 · Indexed by Pub-AI: 2026-10-30 UTC
Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information
First announced by arXiv: 2026-10-30 · Original record ↗
- arxiv
- 2610.40030
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-10-30 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40030v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Memory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence. We argue that both are governed by one quantity, the value of information, and propose to estimate it from archived rollouts with a learned world model. We obtain that, when read at the observation, it yields a provable lower bound on the optimal value of the acquiring action, which floors its TD target and attributes this floor to the observation that earned it. Read at the decision, it defines the optimal retrieval set. A pilot retrieval system, to be trained and improved with this signal, solves a memory task with reduced context.