Latent Dynamics · 2026-11-13

Recursive State-Augmented Language Models: Persistent Latent-State Adaptation of Pretrained Language Models

Omer Haq

arXiv:2611.16375PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Recursive State-Augmented Language Models (Recursive SALM) maintain a persistent latent state that evolves after each observation and generates adaptations of a frozen pretrained language model via a modular read–transition–write architecture. In deterministic experiments using frozen GPT-2 small, latent-gated LoRA matches a full-context LoRA baseline on a four-turn hypothesis-inference task. Latent-state interventions show predictions depend on the state when earlier evidence is absent from the decoder context.

Why it matters

The authors address a context-length limitation by using a fixed-size latent state to carry earlier evidence across turns. Their controlled study isolates whether persistent latent-state information is actually used (via state interventions) and compares candidate ranking vs exact generation when decoder context lacks earlier evidence.

Method

  • Model sequential observations with a persistent latent state z_t and a modular read–transition–write architecture: read encodes completed observations, transition updates latent dynamics, write generates an adaptation used by the decoder for prediction.
  • Provide deterministic and stochastic formulations; the experiments evaluate only deterministic SALM with a frozen GPT-2 small backbone.
  • Use state interventions (zeroing/shuffling the latent state) and task evaluations that exclude earlier evidence from the decoder context to test reliance on persistent state.

Limitation

The experiments use GPT-2 small, a 64-dimensional state, 256 training episodes per task, and three model seeds on fixed 64-episode test sets; they evaluate only deterministic SALM and leave longer-horizon retention, overwrite/interference, and stochastic SALM unaddressed.

Abstract (from arXiv)

Language models rely on finite token contexts, limiting access to earlier interactions. We introduce Recursive State-Augmented Language Models (Recursive SALM), a framework in which a persistent latent state evolves with each observation and generates adaptations of a pretrained language model. A modular read-transition-write architecture separates observation encoding, latent dynamics, and predictor adaptation. We develop deterministic and stochastic formulations. The experiments evaluate only deterministic SALM with frozen GPT-2 small and exclude earlier evidence from its decoder context. On a four-turn hypothesis-inference task, latent-gated LoRA reaches 89.1% accuracy, comparable to 88.0% for a trained full-context LoRA baseline, with earlier evidence available solely through a 64-dimensional state. Zeroing or shuffling the state substantially reduces accuracy. In a two-turn task with codewords unseen in training, several interfaces improve candidate ranking over chance, but rarely generate the exact codeword under the evaluated training schedule. These experiments show that persistent latent-conditioned adaptation can support sequential inference when earlier evidence is absent from the decoder context.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-12 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Recursive State-Augmented Language Models: Persistent Latent-State Adaptation of Pretrained Language Models

Omer Haq

Preprint posted: 2026-11-12 · Original record ↗

arxiv
2611.16375
Explicit source versions (1)
  • v1 · 2026-11-12
  1. Preprint available

    Source date: 2026-11-12 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (3)
Abstract · arxiv

Language models rely on finite token contexts, limiting access to earlier interactions. We introduce Recursive State-Augmented Language Models (Recursive SALM), a framework in which a persistent latent state evolves with each observation and generates adaptations of a pretrained language model. A modular read-transition-write architecture separates observation encoding, latent dynamics, and predictor adaptation. We develop deterministic and stochastic formulations. The experiments evaluate only deterministic SALM with frozen GPT-2 small and exclude earlier evidence from its decoder context. On a four-turn hypothesis-inference task, latent-gated LoRA reaches 89.1% accuracy, comparable to 88.0% for a trained full-context LoRA baseline, with earlier evidence available solely through a 64-dimensional state. Zeroing or shuffling the state substantially reduces accuracy. In a two-turn task with codewords unseen in training, several interfaces improve candidate ranking over chance, but rarely generate the exact codeword under the evaluated training schedule. These experiments show that persistent latent-conditioned adaptation can support sequential inference when earlier evidence is absent from the decoder context.