Robotics · 2026-10-20

Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation

Chongkai Gao, Wenlong Huang, Haonan Chen, Lin Shao, Yilun Du, Jiajun Wu, Ruohan Zhang, Li Fei-Fei

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DreamCan is a hierarchical policy for long-horizon robot manipulation that plans with sparse visual subgoals grounded by reachability and task utility. From action-free robot video datasets, it learns dataset-supported reachability and task progress, uses dynamic programming to select subgoal sequences without manual annotations, fine-tunes a pretrained video model to generate subgoals, and executes them with an independently trained goal-conditioned low-level policy.

Why it matters

The authors address the gap between plausible visual futures and executable robot behaviors by planning only a small set of intermediate visual outcomes. They avoid dense trajectory generation and instead learn subgoals that trade off reachability and task utility, then demonstrate hierarchical execution, test-time search/language steering, scaling with suboptimal data, and real-world performance on contact-rich bimanual tasks.

Method

Learn dataset-supported reachability V_R from action-free videos using goal-conditioned offline RL with sparse rewards; learn task utility U(s_t) as task progress from RoboMeter (with LoRA) or VLM/SAM for spatial tasks.
Automatically annotate each video with sparse visual subgoal sequences via dynamic programming that penalizes difficult transitions (reachability cost) and uses a spacing prior while maximizing utility progress.
Fine-tune a pretrained video generation model (Wan2.2-TI2V-5B) to output a sequence of latent-compressed visual subgoals, then execute them with an independently trained goal-conditioned low-level policy (0.1B parameters).

Limitation

The authors state that a dense video planner can suffer from severe hallucinations such as object disappearance and deformation on long-horizon tasks.

Abstract (from arXiv)

Video models provide rich priors for generalizable robotic manipulation, but plausible visual futures are not always executable. We introduce DreamCan, a hierarchical policy that grounds video models through sparse visual planning: a high-level planner adapted from a pre-trained video model and a low-level visuomotor policy. Rather than predicting dense trajectories, the high-level planner generates sparse visual subgoals that balance reachability and task utility. From offline, action-free robot videos, we estimate these quantities and use dynamic programming to select subgoal sequences without manual annotations. The resulting subgoals are more reliably attained than uniformly spaced alternatives, and the full system outperforms language, dense-video, and uniform-subgoal planners, reaching high planning success rates on Language-Table tasks and 91.2% success on LIBERO-Long with a 0.1B-parameter low-level policy. Additional suboptimal videos and test-time compute further improve success. The system also performs challenging real-world bimanual tasks, including box folding and packing with unseen object combinations. More results are available at https://dreamcan.github.io/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-18 · Indexed by Pub-AI: 2026-10-20 UTC

preprint