Driving · 2026-10-28
Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction
Taeyoung Kim, Junyoung Sung, Heeseung Kwon, Minseop Kim, Jinwoo Shin
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
World Action Models adapt pretrained video generative models for robotic control. The authors show that redistributing visual supervision toward dynamic regions improves policy performance, then introduce Motion-Guided Asymmetric Prediction (MoGAP): an auxiliary training-only path that allocates higher diffusion noise to dynamic regions (from latent motion scores) and uses static regions as context, improving success on GR-1 and beyond.
Why it matters
The paper targets a key adaptation bottleneck in WAMs: visual prediction supervision inherited from video pretraining treats regions uniformly, even though robot control depends more on dynamic regions (motion and object interactions). MoGAP provides a way to tailor this supervision within the pretrained diffusion backbone, without external teachers or extra modalities, yielding consistent policy gains and improved real-world adaptation to task-relevant changes.
Method
- Motivating study: compute latent motion scores from temporal differences in clean video latents and redistribute the video visual supervision toward higher-motion regions, while keeping action prediction unchanged.
- MoGAP task: build an asymmetric dual-timestep assignment using motion-guided latent motion masks; dynamic tokens receive more corrupted noise while static tokens are less corrupted; apply flow-matching loss only to moving tokens.
- Integrate MoGAP as a training-only auxiliary prediction path within the shared video diffusion backbone, with an asymmetric attention mask so original WAM inference tokens cannot attend to MoGAP tokens; omit MoGAP tokens at inference.
Abstract (from arXiv)
World Action Models (WAMs) leverage pretrained video generative models for robotic control by jointly predicting future visual observations and actions. The visual prediction objective plays a key role in adapting pretrained video representations to the target robot domain. Yet standard visual prediction inherited from video pretraining treats all visual regions uniformly, despite their differing relevance to robotic control. We therefore investigate the spatial allocation of visual supervision in WAMs and show that focusing supervision on dynamic regions can substantially improve policy performance. Building on this finding, we introduce Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction. MoGAP allocates noise asymmetrically across regions, training the model to reconstruct more heavily corrupted dynamic regions using less corrupted static regions as context. MoGAP is trained alongside standard video-action prediction through a training-only auxiliary prediction path within the shared video diffusion backbone. It uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities. Across robot manipulation benchmarks and WAM architectures, MoGAP consistently improves policy performance, increasing success rates on the GR-1 tabletop benchmark from 38.8% to 47.5% for Fast-WAM (Wan2.2-5B) and from 52.3% to 62.3% for Cosmos3-Nano.
Related papers
- Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
- Video Prediction Policy 2: Predict Better, Act Better
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
- Native Action-Prior Learning from Videos for World Action Models
- RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC
Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction
Preprint posted: 2026-10-27 · Original record ↗
- arxiv
- 2610.36624
Explicit source versions (1)
- v1 · 2026-10-27
- Preprint available
Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.36624v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World Action Models (WAMs) leverage pretrained video generative models for robotic control by jointly predicting future visual observations and actions. The visual prediction objective plays a key role in adapting pretrained video representations to the target robot domain. Yet standard visual prediction inherited from video pretraining treats all visual regions uniformly, despite their differing relevance to robotic control. We therefore investigate the spatial allocation of visual supervision in WAMs and show that focusing supervision on dynamic regions can substantially improve policy performance. Building on this finding, we introduce Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction. MoGAP allocates noise asymmetrically across regions, training the model to reconstruct more heavily corrupted dynamic regions using less corrupted static regions as context. MoGAP is trained alongside standard video-action prediction through a training-only auxiliary prediction path within the shared video diffusion backbone. It uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities. Across robot manipulation benchmarks and WAM architectures, MoGAP consistently improves policy performance, increasing success rates on the GR-1 tabletop benchmark from 38.8% to 47.5% for Fast-WAM (Wan2.2-5B) and from 52.3% to 62.3% for Cosmos3-Nano.