Robotics · 2026-10-20

Vector-Quantized World Model: Latent Feedback Motion Planning for Robot Manipulation

Xuezhou Xu, Chung-Pang Wang, Zhenzhen Li, Benjamin Riviere

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

VQ-WM is a latent feedback motion planning framework for long-horizon robot manipulation. It combines discrete search over vector-quantized latent states with a continuous visual-feedback controller that sequentially reaches planned targets. State vector quantization is based on first-passage time (controller calls to enter a neighborhood), aligning the planning abstraction with controller reachability. The authors report 100% success on physical Tower of Hanoi with only short-play data.

Why it matters

The authors address long-horizon manipulation failures from compounding prediction errors, search complexity, and plans that are hard for the controller to execute. By quantizing latent states using controller-dependent first-passage time and planning with a discrete latent predictor, the method is designed to produce controller-reachable targets. The paper includes real-hardware Tower of Hanoi results and simulation benchmarks where it reports large gains over world-model baselines.

Method

Build a discrete planning space by vector-quantizing latent states into a codebook; quantization uses first-passage time (number of controller calls until the controller first enters a neighborhood of the target).
Plan by graph search (breadth-first search) for a shortest sequence of vector-quantized states using a discrete latent transition predictor P_Q.
Execute the plan with a goal-conditioned continuous controller π using visual feedback, allocating controller calls per planned transition based on estimated first-passage time.

Limitation

In the non-expert dataset setting, even when V-JEPA 2-AC is given manually specified subgoal images (V-JEPA 2-AC*), the authors report it still struggles to execute full H15 solutions reliably, indicating difficulty discovering intermediate moves on its own.

Abstract (from arXiv)

Visual world models have shown strong potential for robot planning, yet long-horizon manipulation remains challenging due to compounding prediction errors, large search complexity, and latent plans that the low-level controller cannot reliably reach. In this work, we introduce Vector-Quantized World Model (VQ-WM), a latent feedback motion planning framework that integrates discrete long-horizon search with continuous visual feedback control: a discrete latent predictor models transitions between vector-quantized latent states, allowing the planner to look far ahead and find a shortest path to the goal through discrete search and a continuous learned controller sequentially reaches the planned states through visual feedback, correcting deviations to mitigate compounding errors. Whereas most vector quantization uses Euclidean distance, we use a novel estimator of the first-passage-time as a quasi-metric, which aligns the planning abstraction with the controller's capabilities. We evaluate on the physical robot manipulation of the Tower of Hanoi puzzle: whereas existing state-of-the-art foundation models (VLAs, WAMs) only succeed when the solution trace in their training dataset, our method achieves 100\% success rate with only short-play data, demonstrate its ability to compose long-horizon solutions beyond the short demonstrated trajectories and execute them reliably. Our solution also outperforms strongest world-models by 13.3--85.3 percentage points on simulation benchmarks spanning manipulation and locomotion, demonstrating consistent progress and new capabilities.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC

preprint