Video World Models · 2026-10-09

Learning to Retrieve: Internalizing Memory Retrieval for Video World Models

JiaKui Hu, Tailai Chen, Yuqi Pan, Xuerui Qiu, Jialun Liu, Xiao Cao, Zhenxin Zhu, Guang Chen, Hangjun Ye, Bing Wang, Yanye Lu

arXiv:2610.11444 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Learning-to-Retrieve (L2R), which internalizes memory retrieval inside a video diffusion transformer for camera-conditioned video world models. L2R repurposes a persistent internal state as memory, using a camera-conditioned retrieval gate (what to retrieve) and a retrieval trigger supervised by a 3D re-visibility signal (when to retrieve). This aims to improve long-term scene consistency without an external memory bank or 3D conditions.

Why it matters

Scene-level consistency upon camera revisits is a core requirement for explorable video world models. Prior work often relies on external or auxiliary memory retrieval paths, which the authors argue can mismatch the model’s internal generative dynamics. By internalizing retrieval within the generation model, L2R targets improved long-horizon consistency while preserving video quality and camera controllability.

Method

Internalized memory retrieval: repurpose the model’s persistent internal state as historical memory within the video diffusion transformer.
Two learned mechanisms: a camera-conditioned retrieval gate selects what historical information to access; a retrieval trigger determines when to initiate retrieval.
3D re-visibility supervision: tokens are labeled retrieval-relevant when previously observed 3D points exit the frustum (or are occluded) and later reappear in the current view.

Limitation

Future research could focus on the issue of memory retrieval in dynamic scenes. Optimizing retrieval triggers for dynamic scenes using more accurate geometric or external supervision represents a viable path forward.

Abstract (from arXiv)

Video world models aim to generate explorable, 3D-consistent scene videos conditioned on camera trajectories. Existing approaches often rely on external memory systems that explicitly retrieve previously observed content to mitigate scene drift during long-horizon generation. However, these auxiliary memory pathways operate outside the model's internal generative dynamics, preventing the model from intrinsically learning when and what historical information should be retrieved. We propose to internalize memory retrieval into the generation process, allowing retrieval to emerge as an intrinsic behavior of the video world model rather than relying on an external memory system. Based on this principle, we introduce \textbf{Learning-to-Retrieve (L2R)}, which repurposes the model's persistent internal state as a memory for historical context. A camera-conditioned retrieval gate selectively accesses relevant historical information from this state, determining \textit{what to retrieve}, while a retrieval trigger determines \textit{when to retrieve}. We further supervise the trigger with a 3D re-visibility signal, activating retrieval when previously observed content re-enters the current view while otherwise preserving the existing context. Together, these components enable the model to intrinsically acquire memory retrieval behavior and incorporate relevant historical observations into generation without a separate retrieval pathway. Across multiple base models and camera-revisit benchmarks, L2R improves long-term scene consistency while eliminating the need for an external memory bank or 3D conditions. https://jkhu29.github.io/l2r

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC

preprint