Video World Models · 2026-10-02
Memorizon: Training World Models Beyond Their Context Window
Tingting Liao, Xuezhi Liang, Hao Li, Guangyi Liu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Memorizon trains streaming world models on long revisit supervision without paying for long attention. Each sample spans any length but only its last k chunks are scored; unscored history is not tokenized. For each scored chunk, the model retrieves top-K prior latents by camera co-visibility and shares the union as a bounded memory bank (≤ kK entries). The bank increases step cost once; longer spans then add little cost and improve revisit consistency over sliding windows.
Why it matters
It targets the mismatch between training context windows (often tens of seconds) and deployment horizons (minutes) that break revisit supervision. Memorizon decouples long-span supervision from long-context attention by retrieving co-visible latents into a shared bounded bank, enabling models to learn to keep revisited places consistent even when the first visit lies far outside the attention window.
Method
- Training samples draw a span length per sample but score only the last k scored chunks; earlier history is never tokenized.
- Each scored chunk retrieves its own top-K prior latents by camera co-visibility; the union of these requests forms a shared memory bank capped at kK, bounded independently of span length.
- Attention uses a causal sliding window across chunks plus per-chunk masks so each chunk attends to its bounded recent context and its own retrieved bank entries; rollout reads the same layout one chunk at a time.
Limitation
Longer spans cost some image quality; e.g., the bank levels off and longer than certain span lengths gives no further gain, while quality degrades with span.
Abstract (from arXiv)
Streaming world models should render a place consistently across repeated visits. Directly supervising such revisits requires training samples that capture both visits, often spanning minutes. Yet dense attention over the full span incurs quadratic costs, making long-span supervision expensive. Memorizon breaks this coupling: long spans are needed for supervision, but not for attention, since the two visits can share a forward pass without including every intervening frame. A training sample covers a span of any length but is scored only on its last $k$ chunks. Instead of tokenizing the history before them, each scored chunk retrieves its own top-$K$ latents by camera co-visibility, and the union of these requests forms a shared bank. The bank is bounded by $kK$, so the sequence stays bounded however long the span; at the shortest span the recipe is exactly conventional training. Adding the bank raises the cost of a step once; beyond that, a longer span costs little, and going from 100 to 400 s adds 12% to the step time. Against a sliding-window baseline, retrieval raises revisit consistency on every split, and a span long enough to reach the first visit of each return adds a further 24% to 30%, at some cost in image quality; beyond that span, more length no longer helps. Filling the bank from another episode lowers revisit correlation by 83%, so the model uses what it retrieves. Project page: https://tingtingliao.github.io/memorizon
Related papers
- LongForcing: Scaling Temporal Manifold Matching for Long-Horizon World Rollout
- HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
- Frechet Forcing: Training Long-Horizon Streaming World Models from Short Clips
- Learning to Retrieve: Internalizing Memory Retrieval for Video World Models
- History Forcing: Distilling History-Dependent Continuations for Long-Horizon Video World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
Memorizon: Training World Models Beyond Their Context Window
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00544
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Streaming world models should render a place consistently across repeated visits. Directly supervising such revisits requires training samples that capture both visits, often spanning minutes. Yet dense attention over the full span incurs quadratic costs, making long-span supervision expensive. Memorizon breaks this coupling: long spans are needed for supervision, but not for attention, since the two visits can share a forward pass without including every intervening frame. A training sample covers a span of any length but is scored only on its last $k$ chunks. Instead of tokenizing the history before them, each scored chunk retrieves its own top-$K$ latents by camera co-visibility, and the union of these requests forms a shared bank. The bank is bounded by $kK$, so the sequence stays bounded however long the span; at the shortest span the recipe is exactly conventional training. Adding the bank raises the cost of a step once; beyond that, a longer span costs little, and going from 100 to 400 s adds 12% to the step time. Against a sliding-window baseline, retrieval raises revisit consistency on every split, and a span long enough to reach the first visit of each return adds a further 24% to 30%, at some cost in image quality; beyond that span, more length no longer helps. Filling the bank from another episode lowers revisit correlation by 83%, so the model uses what it retrieves. Project page: https://tingtingliao.github.io/memorizon
