Robotics · 2026-10-23
Self-Distilling Future Video Representations for World Action Models
Huiwon Jang, Suhyeok Jang, Hyeonmo Kang, Seungku Kim, Jinwoo Shin, Sihyun Yu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose FRED, a framework to improve World Action Models (WAMs) by training the intermediate future video representations (from a single denoising step) using self-distillation. FRED uses a momentum teacher to align noisy-future representations with targets extracted from noisy ground-truth future observations, and conditions on a short history for temporal context. Across real-world manipulation tasks, FRED outperforms competing WAMs, especially on long-horizon/out-of-domain generalization and human-to-robot transfer.
Why it matters
In single-step WAMs, action prediction depends on the quality of intermediate future representations. The paper targets this dependency directly by reducing the gap between representations from noise vs clean future observations, using self-distillation and short-history conditioning. The reported gains on zero-shot generalization and human-to-robot transfer suggest that improving one-step future representations can translate into better control.
Method
- Treats WAM’s single denoising step as a future representation generator: h = fθ(ε|ot, c), where intermediate hidden states condition action decoding.
- FRED trains the representation generator with self-distillation: a momentum (EMA) teacher extracts target representations from noisy ground-truth future observations at the same intermediate layer; the student matches patch-wise distributions via KL divergence.
- Adds temporal context by conditioning both student and teacher on a short history of observations (H-frame history) instead of only the current frame.
Limitation
A potential limitation of FRED is its reliance on the language understanding capabilities of the underlying video diffusion backbone, as the training objective focuses on future representation learning and does not provide additional supervision for language grounding.
Abstract (from arXiv)
World Action Models (WAMs) have emerged as a promising approach to learning generalist robot policies using rich spatiotemporal priors from pre-trained video diffusion models. A recent line of WAMs improves inference efficiency by conditioning a separate action decoder on intermediate future video representations extracted from a single denoising step, avoiding costly iterative denoising required for video generation. This design makes action prediction depend directly on the quality of these future representations. In this paper, we introduce Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step. Specifically, FRED uses self-distillation to predict future representations from noise, using representations extracted from ground-truth future observations as targets. We further condition the video model on historical observations, providing temporal context for predicting future representations better. Across diverse real-world manipulation scenarios, FRED consistently outperforms competing WAMs. It achieves particularly large gains on long-horizon and out-of-domain tasks under zero-shot environment generalization, and substantially improves human-to-robot skill transfer, reaching 50.0% task success on tasks absent from robot training data and demonstrated only in human videos, compared with 19.4% for the best competing WAM.
Related papers
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
- Box-WAM: Box Trajectories as Compact State Representations for World Action Models
- Native Action-Prior Learning from Videos for World Action Models
- Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-23 UTC
Self-Distilling Future Video Representations for World Action Models
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.30320
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World Action Models (WAMs) have emerged as a promising approach to learning generalist robot policies using rich spatiotemporal priors from pre-trained video diffusion models. A recent line of WAMs improves inference efficiency by conditioning a separate action decoder on intermediate future video representations extracted from a single denoising step, avoiding costly iterative denoising required for video generation. This design makes action prediction depend directly on the quality of these future representations. In this paper, we introduce Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step. Specifically, FRED uses self-distillation to predict future representations from noise, using representations extracted from ground-truth future observations as targets. We further condition the video model on historical observations, providing temporal context for predicting future representations better. Across diverse real-world manipulation scenarios, FRED consistently outperforms competing WAMs. It achieves particularly large gains on long-horizon and out-of-domain tasks under zero-shot environment generalization, and substantially improves human-to-robot skill transfer, reaching 50.0% task success on tasks absent from robot training data and demonstrated only in human videos, compared with 19.4% for the best competing WAM.