Robotics · 2026-10-02
MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov, Alexey K. Kovalev
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
MIKASA-Robo-VLA is a benchmark of 90 language-conditioned robotic manipulation tasks designed to evaluate memory in long-horizon VLA policies. Most tasks hide the cue an action depends on; 10 are reactive controls. The suite provides an environment-based “information gap” and releases 22,500 oracle trajectories in RLDS and LeRobotDataset v3. A reference π0.5 baseline achieves 0.211 ± 0.044 mean success on 14 tasks.
Why it matters
The authors address a limitation of VLA evaluation that often uses only recent frames by creating tasks where critical information disappears before dependent actions. By parameterizing both the type of memory needed and the cue-to-action information gap (including gaps exceeding 16 steps), plus releasing demonstrations and a common evaluation protocol, the benchmark targets which policies must retain information, and for how long.
Method
- Create 90 ManiSkill 3 language-conditioned manipulation tasks across 10 memory types, with 10 reactive controls where the relevant cue stays visible.
- Define an environment “information gap” as the interval between the last availability of a task-relevant cue and the start of the dependent action phase (70 tasks have determined gaps).
- Provide a reference evaluation protocol and fine-tune π0.5 on a subset with current images + proprioception only (no observation history or explicit memory module).
Limitation
The reference results cover one training run and one checkpoint on 14 of 90 tasks; baseline failure can reflect memory, motor error, or distribution shift due to open-loop 8-action chunks and training on successful demonstrations with limited recovery. Phase timing captures explicit information gaps, not all policy-dependent retention demands, and does not rule out shortcuts.
Abstract (from arXiv)
Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses language only in a representative VLA subset. Here every task provides an instruction, while memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every memory-dependent task still requires memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3. A reference $\pi_{0.5}$ baseline with current images and proprioception, but no observation history or explicit memory module, is fine-tuned on 14 tasks and achieves 0.211 $\pm$ 0.044 mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the memory types represented in that subset. Project page: https://mikasarobo.github.io/
Related papers
- Fast-VLA: You Only Think Once for Robot Manipulation
- EvoMem-VLA: State-Evolution Memory for Long-Horizon Robot Manipulation
- Plan First, Act Later: Streaming Reasoning for Executable Memory in Robot Control
- VeriVLA: Adaptive Test-Time Verification for Flow-Based VLAs with Latent World Models
- PaceVLA: Adaptive Temporal Action Quantization in Vision-Language-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00604
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses language only in a representative VLA subset. Here every task provides an instruction, while memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every memory-dependent task still requires memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3. A reference $π_{0.5}$ baseline with current images and proprioception, but no observation history or explicit memory module, is fine-tuned on 14 tasks and achieves 0.211 $\pm$ 0.044 mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the memory types represented in that subset. Project page: https://mikasarobo.github.io/