Video World Models · 2026-10-30

Context Compaction for World Models

Hayder Tirmazi

arXiv:2610.38980PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors frame world-model consistency on revisits as a context retrieval problem, separating (1) model-layer context capacity (KV cache/attention) from (2) agent-harness context selection/compaction. They build a room-revisit benchmark and a context-compaction agent harness. They report that context compaction retrieves rooms up to 4× beyond the model context window without retraining, and that SWA can lose most context retrieval when dreaming.

Why it matters

World models often fail to regenerate revisited regions consistently. By decoupling context retrieval from context reconstruction/inference, and by studying both model-layer and agent-harness choices, the work shows that extending effective retrieval beyond the model’s context window can come from the harness (context compaction) rather than only from changing attention/KV cache in the model. This also highlights that some model-layer choices (sliding-window attention) can degrade retrieval when the world model is used for dreaming.

Method

  • Build a benchmark where episodes randomly generate rooms each time, preventing reliance on memorized training-room layouts; score revisits with both a normal context window and a “shuffled” run to separate context retrieval from context inference.
  • Introduce an agent harness that performs context compaction for world models without retraining or modifying the model weights.
  • Evaluate four world-model designs (decoder-only, CED, CED-SWA, and DD) under raw vs compacted contexts, including when the agent dreams (world-model prediction).
Abstract (from arXiv)

World models let observers explore and interact with generated worlds. Recent work shows that when observers revisit a region, current models often fail to regenerate it consistently. We view this as a context retrieval problem. LLM agents manage context retrieval at two layers of their stack. At the model layer, the KV cache design determines how much context fits in memory. At the agent harness layer, harnesses such as Claude Code pick the context sent to the LLM. Prior work on world models has mostly focused only on the model layer. In this paper, we make three contributions. First, we build a benchmark to measure world model context retrieval. It generates new worlds for every episode, preventing a model from recalling rooms using training data. It also separates context retrieval from guesses based on nearby frames. Second, we build an agent harness that performs context compaction for world models, which to our knowledge no prior work has studied. Third, we use our benchmark to study four world model designs. Context compaction lets our world models retrieve rooms up to 4 times beyond their context window without retraining. However, world models with sliding-window attention lose most of their context retrieval when dreaming. Therefore, a dreaming world agent benefits more from extending the reach of its world model using context compaction than switching to sliding window attention at the model layer.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-10-30 · Indexed by Pub-AI: 2026-10-30 UTC

preprint

Context Compaction for World Models

Hayder Tirmazi

First announced by arXiv: 2026-10-30 · Original record ↗

arxiv
2610.38980
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-10-30 · First detected: 2026-10-30 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2610.38980v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

World models let observers explore and interact with generated worlds. Recent work shows that when observers revisit a region, current models often fail to regenerate it consistently. We view this as a context retrieval problem. LLM agents manage context retrieval at two layers of their stack. At the model layer, the KV cache design determines how much context fits in memory. At the agent harness layer, harnesses such as Claude Code pick the context sent to the LLM. Prior work on world models has mostly focused only on the model layer. In this paper, we make three contributions. First, we build a benchmark to measure world model context retrieval. It generates new worlds for every episode, preventing a model from recalling rooms using training data. It also separates context retrieval from guesses based on nearby frames. Second, we build an agent harness that performs context compaction for world models, which to our knowledge no prior work has studied. Third, we use our benchmark to study four world model designs. Context compaction lets our world models retrieve rooms up to 4 times beyond their context window without retraining. However, world models with sliding-window attention lose most of their context retrieval when dreaming. Therefore, a dreaming world agent benefits more from extending the reach of its world model using context compaction than switching to sliding window attention at the model layer.