Latent Dynamics · 2026-10-22
Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books
Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh, George Nigmatulin, Kang Li, Stefan Zohren, Jakob Foerster, Mihai Cucuringu, Anisoara Calinescu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study a message-only 8.2M-parameter Mamba-3 state-space foundation model trained autoregressively on tokenized limit-order-book (LOB) message streams. They report that the model’s recurrent state contains decodable market microstructure dynamics: order-flow imbalance (OFI) is decoded at R^2≈0.59 (far above controls), generalizes to unseen stocks, and is causally steerable via monotonic dose responses.
Why it matters
This paper provides mechanistic interpretability evidence for a state-space foundation model trained natively on real LOB message data. The authors locate key information in the recurrent state (not the residual stream), quantify decodability of microstructure signals, test robustness with controls and forward-chaining cross-validation, and show causal steering/interventional effects—relevant for both scientific understanding and practical trust in model behavior under changing market conditions.
Method
- Train an 8.2M-parameter Mamba-3 state-space model autoregressively on tokenized LOB message streams; the model never receives the order book as input.
- Mechanistic probing: use kernel ridge readouts from the residual stream vs the recurrent state to decode touch, OFI, traded volume, realized volatility, and depth at a fixed boundary; compare vs random-weights twin and book-snapshot baselines.
- Causal steering: write feature-conditional directions (e.g., OFI) into the recurrent state at varying “dose,” then measure shifts in the model’s next-token distribution under teacher forcing (primary) and via market replay (secondary).
Limitation
Its forward information is limited: a linear readout of the frozen state predicts the next 25 messages’ mid-price move at only R^2≈0.04, on par with a readout of the visible book.
Abstract (from arXiv)
Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs). Yet, without understanding their inner workings, research directions are hard to narrow and practical use in real-world trading environments remains uncertain. This study therefore asks what these models actually learn about market microstructure. We find that, similarly to games where sequence models build an internal representation of the board, an 8.2M-parameter Mamba-3 state-space model trained autoregressively on the tokenized message stream of eight large-cap US equities learns an internal representation of the market and its dynamics. In particular, order-flow imbalance, a central driver of short-horizon price moves, is decodable at $R^2 \approx 0.59$, far above controls and baselines. This representation extends to unseen stocks, suggesting that these models learn general representations rather than ticker-specific patterns. The model also generalizes across metrics beyond order-flow imbalance: results hold across traded volume and volatility. Additionally, the representation is causally steerable: pushing the state along a feature's axis shifts the model's next-message beliefs as that feature implies, monotonically in dose, consistent with the emergence of a world model.
Related papers
- Towards Financial World Modeling
- STOCK-JEPA: Prior-Anchored Latent Revision Representation Learning in Equity Markets
- Market-Conditioned JEPA: Learning from Unpredictable Market Shocks
- Pythia: Toward Foundation World Models for Multimodal Time Series
- Correct then Forecast: Observer State-Space Models for Time Series Forecasting
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-22 UTC
Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28753
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs). Yet, without understanding their inner workings, research directions are hard to narrow and practical use in real-world trading environments remains uncertain. This study therefore asks what these models actually learn about market microstructure. We find that, similarly to games where sequence models build an internal representation of the board, an 8.2M-parameter Mamba-3 state-space model trained autoregressively on the tokenized message stream of eight large-cap US equities learns an internal representation of the market and its dynamics. In particular, order-flow imbalance, a central driver of short-horizon price moves, is decodable at $R^2 \approx 0.59$, far above controls and baselines. This representation extends to unseen stocks, suggesting that these models learn general representations rather than ticker-specific patterns. The model also generalizes across metrics beyond order-flow imbalance: results hold across traded volume and volatility. Additionally, the representation is causally steerable: pushing the state along a feature's axis shifts the model's next-message beliefs as that feature implies, monotonically in dose, consistent with the emergence of a world model.
