Driving · 2026-10-14

Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies

Yilin Wu, Rutav Shah, Shuaiyi Huang, Sumedh A. Sontakke, Homanga Bharadhwaj, Andrea Bajcsy, Max Simchowitz, Guanya Shi

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

IMPRINT is a policy-agnostic latent memory framework for vision-language-action (VLA) robot policies that avoids history curation. It compresses multi-stream histories (visual observations, actions, proprioception) into fixed-size latent memory updated incrementally at inference. A two-stage training procedure first supervises memory via multi-stream reconstruction and task-progress estimation, then finetunes end-to-end on action prediction.

Why it matters

Robotic long-term memory is often handled by manually curating what to remember, which fixes a notion of relevance and can omit needed intermediate or sensorimotor information. The authors propose a way to learn memory contents without explicitly constraining relevance, while keeping inference efficient by using a fixed-size recurrent bottleneck for multi-stream latent memory.

Method

Compress multi-stream interaction histories into fixed-size latent memory tokens conditioned on task language; use a frozen causal video tokenizer for visual compression and a learned latent bottleneck (full-attention or recurrent GDN) updated incrementally at inference.
Stage 1 memory bootstrapping: supervise memory with reconstruction of past visual features, actions, and proprioceptive states, plus progress estimation over the episode using policy-independent objectives.
Stage 2 control adaptation: discard the reconstruction decoder, integrate the learned memory into a VLA via a lightweight action-expert adapter (FiLM + cross-attention to memory tokens), and jointly finetune on action prediction.

Abstract (from arXiv)

It is widely appreciated that robotic policies require long-term memory to perform complex tasks. However, directly conditioning policies on all past observations leaves policy learning susceptible to learning spurious shortcuts, and introduces large latency at policy deployment. As such, today's memory-augmented policies curate what to remember, either by retaining only recent frames, selected keyframes, or textual summaries, or by learning latent representations of the visual stream alone. In this work, we propose an alternative approach IMPRINT, a policy-agnostic latent memory framework that augments vision-language-action (VLA) models with multi-stream memory without explicitly constraining what the policy should remember. IMPRINT compresses different streams into a latent memory that is updated incrementally at inference, enabling low latency. To avoid shortcut learning, IMPRINT adopts a two-stage training procedure, first supervising memory to reconstruct multiple streams (observations, actions, proprioception) of past information, as well as estimate current task progress, followed by end-to-end finetuning, jointly with the policy, on action prediction. Across both simulation and real-world tasks demanding memory, IMPRINT improves average success over the base VLA by at least 26%, outperforming popular alternatives, and maintaining lowest inference latency among memory-augmented VLAs. Through careful ablations, we demonstrate the utility of incorporating diverse information streams into robotic memory, validating the advantages of foregoing history curation. Project website: https://yilin-wu98.github.io/imprint_website/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-14 UTC

preprint