Latent Dynamics · 2026-10-14

RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations

Yuanpeng He, Ceyao Zhang, Fangjing Li, Kefei Zhu, Lijian Li, Yueheng Li, Tianjia He, Wenpin Jiao, Zhi Jin, Yaodong Yang, Yuanpei Chen

arXiv:2610.16443 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

RAP-WAM learns world-action models for control by combining predictive rehearsal with action perturbations. It uses a future-supervised action interface and directional query attention, then measures video prediction changes under perturbed actions to derive spatiotemporal credit for visual and action supervision. The method avoids extra region scoring/probes and is trained from a video-pretrained backbone. It improves RoboDojo success rates and achieves strong real-robot task-macro success with agent-refined, task-conditioned policy selection.

Why it matters

The authors target a key gap in world-action model control: deciding how future predictions should guide policy learning. RAP-WAM distinguishes prediction error from action dependence by using action-response signals to allocate supervision across visual locations and future action steps. This can turn demonstrations into representation targets and adaptive training emphasis without extra annotation or deployment-time scoring, and it is evaluated on both simulation and real robot tasks.

Method

Future-supervised action interface: recorded future latents supervise query alignment inside the shared video–action predictor, with directional attention to keep action-to-video paths while enabling predictive-query conditioning.
Predictive rehearsal: adjacent demonstration segments are trained by generating the first chunk’s future video/actions (without gradients) and substituting the second chunk’s initial observation with the last predicted frame, while keeping recorded targets; prediction errors accumulate into a spatial prior.
Response-derived spatiotemporal credit: run two forwards on the same recorded demonstration, perturbing normalized noisy actions; use squared prediction error and squared response change (plus the rehearsal prior) to weight video and action losses over future space and time.

Abstract (from arXiv)

Recent world action models draw on video generation priors to jointly predict future observations and robot actions. Adapting these priors to control requires determining how future predictions should guide policy learning. RAP-WAM combines predictive Rehearsal and Action Perturbations (RAP) with future-supervised queries and Coupled Action Projection for policy learning. Predictive rehearsal trains on model-generated inputs while retaining recorded targets, and accumulates recurring errors into a spatial prior. We perturb actions and measure changes in video predictions. Action-response-based spatiotemporal credit assignment combines these responses, prediction error, and the rehearsal prior to allocate visual supervision; spatially aggregating the responses determines supervision intensity across future action steps. No separate scoring network or deployment-time probe is required. The base learns from a video-pretrained backbone without intermediate pretraining on external embodied data. We evaluate the fixed base and its agent-refined, task-conditioned system. On 42 RoboDojo tasks, their average success rates are 12.85% and 15.62%, with Scores of 17.86 and 20.59. Across 15 real-robot tasks, the complete system achieves 89.00% task-macro-average success, exceeding the strongest of five baselines by 37.67 percentage points. Across 315 fixed-output simulation windows, 45.21% of response-top20 positions fall outside error-top20, revealing information beyond prediction error.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-14 UTC

preprint