Robotics · 2026-10-14

RM-WAM: Reliable Memory-Augmented World Action Models for Long-Horizon Robot Manipulation

Cui Miao, Junhao Yu, Ping Liu, Ning Yang, Xuhua Chen, He Zheng, Xiaodong Wang, Tao Zhang

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

RM-WAM is a memory-augmented World Action Model for long-horizon robot manipulation that treats memory as a context-dependent predictive resource. It uses complementary short-term visual memory and latent event memory, and learns a counterfactual utility estimate to gate how much memory to inject into the WAM backbone for both future world prediction and action generation.

Why it matters

The authors argue that directly using historical observations can be stale, stage-mismatched, or error-contaminated, distorting world prediction and accumulating closed-loop errors. RM-WAM explicitly models memory utility with counterfactual supervision and uses a utility-aware gating mechanism to inject reliable memory only, aiming to improve long-horizon decision making under partial observability.

Method

Dual-scale history: short-term visual memory (recent frames compressed into spatial patch-token sequences) plus latent event memory (transition candidates sampled every 16 timesteps, encoded in latent space).
Counterfactual memory learning: vary accessible history while keeping current observation/instruction/proprioception and prediction targets fixed, and assign soft utility targets to supervise memory utility.
Utility-aware gating: learned gates adaptively control memory injection into the WAM video backbone; the video branch and action branch are combined via a Mixture-of-Transformers (MoT).

Abstract (from arXiv)

World Action Models (WAMs) improve robotic manipulation by jointly predicting future visual states and actions, yet most existing methods make decisions primarily from the current observation. This limits their reliability in long-horizon tasks, where visually similar states may correspond to different interaction histories and task stages. Moreover, directly incorporating historical observations can introduce stale, stage-mismatched, or error-contaminated information, leading to inaccurate world prediction and accumulated closed-loop errors. We propose RM-WAM, a reliable memory-augmented World Action Model that treats memory as a context-dependent predictive resource rather than universally useful historical context. RM-WAM represents history using complementary short-term visual state memory and latent event memory, which respectively capture recent interaction dynamics and longer-term state transitions. A counterfactual memory learning scheme varies the accessible history while keeping the current inputs and prediction targets fixed, enabling the model to learn a context-dependent utility estimate for historical memory. Based on this estimate, a utility-aware gating mechanism adaptively controls memory injection into the world-action backbone. Experiments demonstrate that RM-WAM highlights the importance of reliable and selective memory utilization in long-horizon tasks.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC

preprint