JEPA · 2026-10-02
Heavy-Tailed Memory Traces in Long-Horizon Language Agents
Xinyuan Song, Zekun Cai
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors audit how long-horizon language agents’ external memory gets used under finite context and repeated retrieval, finding reproducible core–tail concentration with policy-dependent tail shapes. They propose Core–Tail World Model (CTWM), a rank-based memory controller that allocates prompt budget using an exponent τ while retaining a summarized tail. CTWM reduces prompt tokens and tail prediction error in paired comparisons across graph and text benchmarks.
Why it matters
Beyond task success and token cost, this work emphasizes the distribution of attention induced by memory backends. The authors show that finite retrieval can create core–tail concentration and that, under semantic policies, tail structure relates to prediction quality. CTWM turns the audited memory-trace shape into a tunable allocation rule for more token-efficient agent memory world models.
Method
- Tail audit: the authors use a conservative Clauset-style audit (with log-normal and truncated-power-law alternatives) and analyze core–tail memory access under random-walk vs semantic LLM policies.
- CTWM: a rank-based core–tail memory controller that allocates prompt budget by rank with exponent τ, allocating more mass to high-rank core and summarizing a tail rather than dropping it.
- Evaluation: paired comparisons using actual API-reported prompt tokens per step and coverage/“tail error” metrics, plus ablations separating allocation vs serialization effects.
Limitation
The authors report that Clauset α-hat is unstable under the τ sweep due to automatic cutoff instability, so α-hat is not used as the control metric.
Abstract (from arXiv)
Long-horizon language agents increasingly rely on external memory as a frozen world model, yet current memory systems are usually judged only by task success or token cost. We argue that the missing object is the shape of memory use: under finite context and repeated retrieval, agent memory can concentrate on a small core while leaving rare states in a long tail where prediction errors accumulate. We study this effect through a conservative tail audit and find that concentration is reproducible but policy-dependent. Random-walk agents produce log-normal-compatible retrieval artifacts, whereas semantic LLM policies yield the strongest truncated-power-law-compatible core--tail traces. Motivated by this audit, we propose Core--Tail World Model (CTWM), a rank-based memory controller that allocates prompt budget with a single exponent $\tau$ while retaining a summarized tail. On Synthetic Graph World, CTWM preserves full state and transition coverage, reduces prompt tokens by 5.9%, and lowers bottom-half tail prediction error by 13.6% relative to a graph-memory baseline. The same paired comparison gives consistent token savings on ALFWorld and a 24.48% token reduction on LongMemEval with aggregate accuracy parity. These results suggest that heavy-tailed memory traces are not only a diagnostic of finite retrieval, but also a practical control signal for token-efficient agent world models.
Related papers
- MemoWM: How World Models Change What Agents Need to Remember
- Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
- Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
- Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards
- MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-07-09 · Indexed by Pub-AI: 2026-10-02 UTC
Heavy-Tailed Memory Traces in Long-Horizon Language Agents
Preprint posted: 2026-07-09 · Original record ↗
- arxiv
- 2610.00010
Explicit source versions (1)
- v1 · 2026-07-09
- Preprint available
Source date: 2026-07-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Long-horizon language agents increasingly rely on external memory as a frozen world model, yet current memory systems are usually judged only by task success or token cost. We argue that the missing object is the shape of memory use: under finite context and repeated retrieval, agent memory can concentrate on a small core while leaving rare states in a long tail where prediction errors accumulate. We study this effect through a conservative tail audit and find that concentration is reproducible but policy-dependent. Random-walk agents produce log-normal-compatible retrieval artifacts, whereas semantic LLM policies yield the strongest truncated-power-law-compatible core--tail traces. Motivated by this audit, we propose Core--Tail World Model (CTWM), a rank-based memory controller that allocates prompt budget with a single exponent $τ$ while retaining a summarized tail. On Synthetic Graph World, CTWM preserves full state and transition coverage, reduces prompt tokens by 5.9%, and lowers bottom-half tail prediction error by 13.6% relative to a graph-memory baseline. The same paired comparison gives consistent token savings on ALFWorld and a 24.48% token reduction on LongMemEval with aggregate accuracy parity. These results suggest that heavy-tailed memory traces are not only a diagnostic of finite retrieval, but also a practical control signal for token-efficient agent world models.