Driving · 2026-10-05
World Action Modeling with Progressive Visual Planning
Fei Zhang, Zhaochong An, Duncan Frost, Yikai Wang, Pengfei Liu, Ya Zhang, Michal Drozdzal, Amir Bar
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors present ProWAM, a world action model that predicts an ordered sequence of sparse, progress-indexed visual sub-goals alongside action chunks. This avoids inefficient dense video rollouts by caching sub-goal features from a single video-backbone forward pass, then using lightweight action denoising during replanning. The authors report improved out-of-distribution robustness on multiple simulation benchmarks and zero-shot real-world experiments.
Why it matters
Long-horizon visual forecasting is costly when generating dense video rollouts, and simpler “final frame” or “no imagination” approaches lack intermediate progress guidance. ProWAM provides explicit intermediate visual targets via progress-conditioned sparse sub-goals, enabling efficient closed-loop control through sub-goal feature caching and simpler action denoising.
Method
- ProWAM represents long-horizon futures as an ordered sequence of sparse visual sub-goals indexed by normalized progress r∈[0,1], from current observation (r=0) to trajectory endpoint (r=1).
- It uses a vision-action mixture-of-transformers (MoT) with structured attention: a video expert predicts transition latents and progress-conditioned sub-goal latents; an action expert generates action chunks conditioned on cached visual sub-goal features.
- For efficient replanning, the video expert is run once to predict and cache sub-goal key–value features; the action head reuses the cache across action denoising steps, avoiding iterative full-video generation.
Limitation
Manually Specified Inference Schedule: although ProWAM supports variable sub-goal counts and progress values, the inference schedule is manually specified rather than inferred from the current task state or complexity. The authors state that learning to select the number and placement of sub-goals adaptively is promising future work.
Abstract (from arXiv)
World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient. Some recent WAMs address this by predicting a single future frame without generating the full video, but this approach neglects how to progress toward the goal. We present ProWAM, a progressive world action model that jointly predicts actions and an ordered sequence of sparse visual sub-goals, providing explicit visual guidance to anchor action generation throughout task execution. This design scales naturally, as sub-goal prediction can be learned from large-scale action-free videos, allowing the video backbone to offload complex visual planning from the action policy. For efficient action generation, ProWAM executes a single video-backbone forward pass to cache sparse sub-goal features, eliminating iterative full-video generation and requiring only lightweight action denoising during replanning. Across extensive evaluations, ProWAM achieves superior out-of-distribution robustness. On simulation benchmarks, it sets new state-of-the-art results on LIBERO-Plus (85.8%) and randomized RoboTwin (75.7%), outperforming the strongest baseline with relative gains of up to +35.9%. On RoboCasa365, ProWAM achieves a 48.1% success rate and 18.2% on the challenging Composite-Unseen split, ranking 4th overall. Crucially, in zero-shot real-world experiments, ProWAM achieves 70.0% success, outperforming the strongest baseline by +15.0 (from 55.0% to 70.0%, a +27.3% relative gain) in novel scenes. These results demonstrate the value of progress-indexed visual foresight for closed-loop control. Our program is in https://sii-ferenas.github.io/ProWAM-page.
Related papers
- 1Step-WAM: Your World Action Model but Faster using 1-Step Action Generation
- RealtimeWAM: How Fast Can I Run My World Action Model?
- OpenWAM: An Open Framework for Composable World-Action Models
- CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-05 UTC
World Action Modeling with Progressive Visual Planning
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.02508
Explicit source versions (1)
- v1 · 2026-10-01
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient. Some recent WAMs address this by predicting a single future frame without generating the full video, but this approach neglects how to progress toward the goal. We present ProWAM, a progressive world action model that jointly predicts actions and an ordered sequence of sparse visual sub-goals, providing explicit visual guidance to anchor action generation throughout task execution. This design scales naturally, as sub-goal prediction can be learned from large-scale action-free videos, allowing the video backbone to offload complex visual planning from the action policy. For efficient action generation, ProWAM executes a single video-backbone forward pass to cache sparse sub-goal features, eliminating iterative full-video generation and requiring only lightweight action denoising during replanning. Across extensive evaluations, ProWAM achieves superior out-of-distribution robustness. On simulation benchmarks, it sets new state-of-the-art results on LIBERO-Plus (85.8%) and randomized RoboTwin (75.7%), outperforming the strongest baseline with relative gains of up to +35.9%. On RoboCasa365, ProWAM achieves a 48.1% success rate and 18.2% on the challenging Composite-Unseen split, ranking 4th overall. Crucially, in zero-shot real-world experiments, ProWAM achieves 70.0% success, outperforming the strongest baseline by +15.0 (from 55.0% to 70.0%, a +27.3% relative gain) in novel scenes. These results demonstrate the value of progress-indexed visual foresight for closed-loop control. Our program is in https://sii-ferenas.github.io/ProWAM-page.