Robotics · 2026-10-14

PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Visual Subgoals and Policy Priors

Yang Liu, Hanchen Cui, Karthik Desingh

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PILOT guides long-horizon latent world-model planning for robot manipulation by generating intermediate visual subgoals with a video model and using a learned policy to initialize CEM for action search. In real UR5e experiments across four tasks, the authors report 68.8% success, versus 55.0% without policy initialization, 13.8% for policy-only execution, and 0.0% for direct final-goal planning. Simulation results report persistent gains on longer horizons.

Why it matters

Long-horizon model-based control is difficult because short-horizon objectives driven only by a distant final goal can get stuck in local minima, and naive action sampling can be inefficient. The authors’ approach separates temporal task progression (via video-predicted subgoals) from local action optimization (via policy-initialized CEM using an action-conditioned world model).

Method

Video prediction model generates temporal visual subgoals (uniformly sampled frames from a predicted video) that guide progression across MPC stages.
A learned policy prior initializes CEM by proposing the first h actions; CEM refines action sequences using an action-conditioned world model and a latent subgoal cost.
Receding-horizon control loop: execute only the first h_exec < h actions from the optimized sequence, observe, and replan for the next active subgoal.

Limitation

PILOT can fail when generated subgoals are unreachable, world-model errors misrank actions, or policy-centered sampling limits exploration. Its fixed subgoal schedule cannot accommodate delayed interactions, which can hinder recovery after a failed grasp. Video generation and online optimization also add latency relative to direct policy execution.

Abstract (from arXiv)

Latent world-model planning has shown promising results in goal-conditioned robotic manipulation, but remains challenging for long-horizon planning due to limited temporal guidance and inefficient action search. A distant visual goal provides little guidance for preparatory actions, while uninformed action sampling makes local optimization inefficient. We propose PILOT (Policy-Initialized Local Optimization with Temporal subgoals), which uses a video model to generate intermediate visual subgoals and a learned policy to initialize the cross-entropy method (CEM). CEM refines candidate action sequences toward each subgoal using an action-conditioned world model in a receding-horizon control loop. In our primary real-robot evaluation across four UR5e manipulation tasks, PILOT achieves 68.8% success, compared with 55.0% without policy initialization, 13.8% for policy-only execution, and 0.0% for direct final-goal planning. Simulation results show that these gains persist over longer horizons: PILOT exceeds direct execution of the same policy by 30 percentage points on long Push-T tasks and by 22 percentage points on LIBERO-10. Policy initialization also accelerates local optimization: on Push-T, CEM matches the cost reached by generic initialization after 10 iterations in only 4 iterations and reduces the final planning cost by 24.1% under the same sampling budget. These results show that combining temporal subgoals with policy-guided action search improves both long-horizon task execution and local planning efficiency. Videos are available on the webpage: https://pilot-planner.github.io/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-14 UTC

preprint