Robotics · 2026-10-22

Plan First, Act Later: Streaming Reasoning for Executable Memory in Robot Control

Qinghang Liu, Conglin Wang, Zongsheng Li, Boyong He, Han Gao, Pengcheng Wang, Tao Sun, Lige Liu, Chang Liu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PAL is a hierarchical memory framework for long-horizon vision-language robot manipulation. It maintains a short-term visual buffer, a semantic-event memory, and an executable subtask_list. A high-level VLM streams observations into memory and selects the current subtask for a low-level VLA policy. Training uses two stages: dense update-timing supervision plus self-labeling with task-grounded verification.

Why it matters

The authors target memory-dependent tasks where success depends on information outside the short observation window. Instead of searching a large memory at execution time, PAL stores future-relevant executable subtasks as evidence becomes available, then retrieves the needed subtask later. The approach is evaluated on RoboMME and real-world manipulation tasks, with reported gains over memory and hierarchical baselines.

Method

Hierarchical streaming memory: short-term visual buffer + semantic-event memory + executable subtask_list; a high-level VLM updates these from observation chunks and selects the current subtask for a low-level VLA.
Executable subtask_list planning: the VLM appends future-relevant subtasks to memory when evidence appears, so execution later can retrieve the stored subtask without inferring it then.
Two-stage training: (1) dense update-timing supervision from sparse annotations via sampled query timesteps; (2) self-labeling with task-grounded verification to enforce correct, non-duplicative updates and match execution rollouts.

Limitation

The authors report that PAL performs poorly on InsertPeg, StopCube, and RouteStick, attributing InsertPeg low success to timeouts (peg not inserted within the allowed horizon), StopCube to fine-control/rapid-reaction requirements outside the main scope, and RouteStick to poor coordination between subtask selection and low-level execution; reducing action-chunk size increased RouteStick success.

Abstract (from arXiv)

Long-horizon robot manipulation requires a robot to remember task-relevant information beyond its short observation window. Existing memory-augmented policies typically retrieve relevant information from history and reason over it at execution time, placing a heavy reasoning burden on the high-level model. To address this challenge, we propose PAL (\textbf{P}lan first, \textbf{A}ct \textbf{L}ater), a hierarchical memory framework for vision-language robot manipulation. It maintains a short-term visual buffer, a semantic-event memory, and an executable \texttt{subtask\_list}. A high-level vision-language model (VLM) converts streaming visual observations into the \texttt{subtask\_list}. At execution time, the VLM selects the currently relevant subtask from the \texttt{subtask\_list} and passes it to a low-level vision-language-action (VLA) policy. To teach the model what to write to memory and when to update it, we introduce a two-stage data generation procedure that first expands sparse update supervision and then uses self-labeling with task-grounded verification to generate training data that better match the state distribution encountered during execution. We evaluate our approach on memory-dependent benchmarks and real-world manipulation tasks. On RoboMME, PAL achieves a \textbf{56.05\%} success rate, outperforming the strongest baseline, FrameSamp-Modul, at 44.51\%. In real-world experiments across four manipulation tasks, \pal{} reaches \textbf{60.83\%} overall success, surpassing MemER (45.00\%) and FrameSamp-Modul (28.33\%). These results show that proactively writing future-relevant subtasks into hierarchical memory improves long-horizon visual reasoning and robot control. Code and videos can be found on our website: https://qinghangliu.github.io/pal-webpage/

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC

preprint