Robotics · 2026-10-08
Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation
Tzu-Yu Chuang, Ching-Hsiang Chang, Yi-Hsiu Lee, Yi-Ting Chen, Min Sun, YuanFu Yang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors argue that generative world-model futures don’t directly optimize the compact task variables needed for control. They introduce Entity-Level Goal Readout, which converts a 3D trace world-model representation into a compact SE(3) terminal goal (object-relative pose with depth-grounded translation). A Pose-Native Executor uses this fixed goal with online pose feedback. On five ManiSkill3 manipulation tasks, the pipeline reaches 79.69% mean success.
Why it matters
The paper targets a practical gap: predicted futures may not yield accurate terminal goals for execution. By training the prediction-to-execution interface to explicitly supervise terminal goal prediction and then measuring goal diagnostics, success, and sensitivity to goal error, the authors show that terminal goal accuracy and its usability for a fixed executor are central when using world models for robotics control.
Method
- Entity-Level Goal Readout: from a 3D trace world-model representation, the system predicts an object-relative terminal pose in SE(3) using an entity branch (terminal rotation) plus a map branch grounded in observed depth (terminal translation).
- Pose-Native Executor: the executor runs at 20 Hz using the fixed episode goal and online object-pose feedback, correcting execution without rerunning the world model.
- Training uses joint readout/trace objectives and DAgger distillation with goal perturbations sampled from measured planner error distributions to train the controller under imperfect goals.
Limitation
A single terminal pose does not explicitly encode trajectory constraints, contact sequences, or articulated and deformable-object states. Planning occurs once per episode, so pose feedback can correct execution but cannot revise an invalidated goal; the system also depends on reliable tracking, depth, and pose estimates, and new goal semantics may require additional supervision.
Abstract (from arXiv)
Generative world models provide rich predictions of how manipulation scenes may evolve toward task objectives, yet those futures do not directly expose the compact task variables required by control. When training supervises future prediction alone, terminal goal accuracy is not an explicit learning objective, even when geometric recovery is available. We present Entity-Level Goal Readout, a learned prediction-to-execution interface that makes the executable terminal goal an explicit output of a 3D trace world model. It combines object-centric pose prediction with translation grounded in observed depth to produce a compact goal in SE(3). A shared Pose-Native Executor consumes this fixed goal with online object-pose feedback for closed-loop control without rerunning the world model. Across five manipulation tasks, the pipeline achieves a mean success rate of 79.69%. Goal diagnostics directly measure terminal goal accuracy, while controlled translation perturbations characterize how execution degrades under goal error. Zero-shot deployment on a Franka arm achieves 73.33% success on nominal StackCube, 66.67% with distractors, and 75.00% on PickPlate with a target unseen during policy training. These results support treating the prediction-to-execution interface as an explicit learned component of world-model planning rather than incidental post-processing in the control pipeline itself. Project page: https://claire0730.github.io/executable-goals/
Related papers
- Achieve What You Imagined: Learning to Align Actions with Visual Plans
- The Planning Limits of Latent World Models
- Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation
- What Limits Cross-Task Generalization in World Action Models?
- Dynamic Manipulation with World-Action Models via Counterfactual Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC
Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.09309
Explicit source versions (1)
- v1 · 2026-10-07
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Generative world models provide rich predictions of how manipulation scenes may evolve toward task objectives, yet those futures do not directly expose the compact task variables required by control. When training supervises future prediction alone, terminal goal accuracy is not an explicit learning objective, even when geometric recovery is available. We present Entity-Level Goal Readout, a learned prediction-to-execution interface that makes the executable terminal goal an explicit output of a 3D trace world model. It combines object-centric pose prediction with translation grounded in observed depth to produce a compact goal in SE(3). A shared Pose-Native Executor consumes this fixed goal with online object-pose feedback for closed-loop control without rerunning the world model. Across five manipulation tasks, the pipeline achieves a mean success rate of 79.69%. Goal diagnostics directly measure terminal goal accuracy, while controlled translation perturbations characterize how execution degrades under goal error. Zero-shot deployment on a Franka arm achieves 73.33% success on nominal StackCube, 66.67% with distractors, and 75.00% on PickPlate with a target unseen during policy training. These results support treating the prediction-to-execution interface as an explicit learned component of world-model planning rather than incidental post-processing in the control pipeline itself. Project page: https://claire0730.github.io/executable-goals/
