Driving · 2026-11-03
MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation
Diqi He, Jinbang Guo, Yijun Shen, Zhiwei Yu, Huan Yang, Dingwen Zhang, Junwei Han, Zhongyuan Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
MOST is an instruction-conditioned, end-to-end policy for long-horizon mobile manipulation that uses policy-internal memory structured around task progress and object-state changes. It compresses multi-view history into temporal tokens, supervises historical and predictive future tokens with subtask and object states during training, and conditions action generation without an external planner or retrieval stage. The authors introduce RoboCasa-Long and report consistent gains over XR-1 and a visual-history-only variant.
Why it matters
The paper argues that long-horizon agents fail not just because observations become partial, but because they must preserve decision-relevant state changes and progress after they leave view. MOST’s memory targets exactly these quantities and evaluates on multi-minute mobile manipulation (RoboCasa-Long), showing improvements across simulation and real-world experiments.
Method
- Policy-internal memory: compress multi-view history into one token per sampled timestep, plus K future queries, forming a single memory sequence.
- Task-state supervision: training-only heads predict each token’s subtask stage and each relevant object’s state; auxiliary heads are removed at inference.
- Action conditioning: paired Memory Expert outputs are fed into Action Expert layers via gated cross-attention, enabling one end-to-end policy without external planner or retrieval.
Limitation
The authors do not use a learned judge for their progress metric construction (they use discrete milestone curves from simulator state or manual scoring), so reported progress metrics are not directly comparable with those computed from PRM-predicted curves.
Abstract (from arXiv)
Long-horizon embodied manipulation requires policies to act on information that is no longer visible. Modified objects may leave the field of view, while completed subtasks continue to constrain future actions. Existing approaches extend temporal context through recurrent state updates, test-time training, or storage and retrieval of past observations. As horizons grow, these mechanisms can still leave the policy to infer task state from long, redundant, and viewpoint-dependent visual experience. We argue that long-horizon agents benefit not merely from remembering more, but from remembering task-relevant state changes and progress. We propose Memory of Object States and Task Progress (MOST), a policy-internal memory framework that compresses multi-view observations into temporal tokens. Historical and predictive future tokens are explicitly supervised with object states and subtask progress, structuring memory around what has changed in the world and what has already been accomplished. The resulting memory directly conditions action generation within a single end-to-end policy, without an external planner or retrieval stage. We further introduce RoboCasa-Long, a benchmark of multi-minute, multi-stage mobile manipulation tasks. Across RoboCasa365, RoboCasa-Long, and real-world robot experiments, MOST consistently outperforms Xiaomi-Robotics-1 (XR-1), the previous state-of-the-art model. It raises average success from 65.9% to 72.1% on RoboCasa365, average maximum task progress from 20.83% to 42.22% on RoboCasa-Long, and real-world success from 6.66% to 43.33%. MOST also outperforms a matched visual-history memory variant, showing that effective long-horizon memory depends not only on how much experience is retained, but on whether it preserves decision-relevant progress and world-state changes.
Related papers
- RM-WAM: Reliable Memory-Augmented World Action Models for Long-Horizon Robot Manipulation
- MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
- EvoMem-VLA: State-Evolution Memory for Long-Horizon Robot Manipulation
- T$^2$Mem: Learning Test-Time Memory for Robotics
- Plan First, Act Later: Streaming Reasoning for Executable Memory in Robot Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.00734
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00734v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Long-horizon embodied manipulation requires policies to act on information that is no longer visible. Modified objects may leave the field of view, while completed subtasks continue to constrain future actions. Existing approaches extend temporal context through recurrent state updates, test-time training, or storage and retrieval of past observations. As horizons grow, these mechanisms can still leave the policy to infer task state from long, redundant, and viewpoint-dependent visual experience. We argue that long-horizon agents benefit not merely from remembering more, but from remembering task-relevant state changes and progress. We propose Memory of Object States and Task Progress (MOST), a policy-internal memory framework that compresses multi-view observations into temporal tokens. Historical and predictive future tokens are explicitly supervised with object states and subtask progress, structuring memory around what has changed in the world and what has already been accomplished. The resulting memory directly conditions action generation within a single end-to-end policy, without an external planner or retrieval stage. We further introduce RoboCasa-Long, a benchmark of multi-minute, multi-stage mobile manipulation tasks. Across RoboCasa365, RoboCasa-Long, and real-world robot experiments, MOST consistently outperforms Xiaomi-Robotics-1 (XR-1), the previous state-of-the-art model. It raises average success from 65.9% to 72.1% on RoboCasa365, average maximum task progress from 20.83% to 42.22% on RoboCasa-Long, and real-world success from 6.66% to 43.33%. MOST also outperforms a matched visual-history memory variant, showing that effective long-horizon memory depends not only on how much experience is retained, but on whether it preserves decision-relevant progress and world-state changes.
