Pub-AI: AI in Science

Video World Models · 2026-09-30

Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

Beomsu Kim, Chieh-Hsin Lai, Bac Nguyen, Amir Bar, Jong Chul Ye, Yuki Mitsufuji

arXiv:2609.34677PDF

Auto-summarized: this summary was generated by a language model from the paper’s text and has not been reviewed by an editor. Check the paper before relying on it. Benchmark numbers appear only after a human has verified them.

TL;DR

The authors propose Future-Aware Recall (FAR), a framework for episodic memory access in world models. FAR learns which past memories to retrieve and which retrieval cues (time, pose, vision, audio) to trust by supervising a future-blind retriever using future-aware predictive utility during training (approximated by negative diffusion prediction loss).

Why it matters

Fixed recall rules (e.g., recency, pose overlap, visual similarity) can be unreliable because cue usefulness varies across environments and queries. FAR provides a principled way to train a retriever to select predictive memories and adaptively weight multiple cues, improving episodic recall and downstream video predictions across three settings.

Method

  • During training, FAR supervises recall relevance using future-aware predictive utility: uθ(C|o_{t+1},Q_t)=log pθ(o_{t+1}|C,Q_t), approximated by negative diffusion prediction loss of the realized future conditioned on candidate memories.
  • FAR’s retriever remains future-blind at inference: it scores candidate memories using retrieval cues Z_{t-1} and query R_t, selecting a compact Top-K context Ct for prediction.
  • FAR learns cue-specific relevance and query-dependent fusion weights (time, pose, vision, audio as available), enabling adaptive cue trust per query.
Abstract (from arXiv)

World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past. Episodic memory preserves past observations for later recall; however, as memory accumulates, it raises a fundamental question: which memories are useful for the current prediction, and which available retrieval cues should be trusted to find them? This is challenging because fixed criteria based on recency, pose overlap, or visual similarity can be unreliable across environments and queries. We propose Future-Aware Recall (FAR), a framework that learns episodic recall from future-aware predictive supervision and adaptive multi-cue scoring. During training, FAR measures predictive utility by the conditional log-likelihood of the realized future given recalled context, approximated by negative diffusion prediction loss, and uses it to train a retriever that remains future-blind at inference. The retriever learns cue-specific relevance and automatically determines which available retrieval cues, such as time, pose, vision, and audio, to trust for each query when selecting memories. Across three complementary settings, FAR outperforms hand-designed recall even with the same retrieval cues, automatically adapts which available cues to trust, and recalls the right history as the world changes. Together, these results establish FAR as a flexible, principled approach to episodic memory access in world models.

Related papers