Driving · 2026-10-06
SUAVE: Unified Video-Action Models via Masked Diffusion
Rhythm Syed, Jean Mercat, Sedrick Keh, Kushal Arora, Paarth Shah, Aykut Onol, Mengchao Zhang, Tony Dear
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SUAVE is a single masked-diffusion transformer that represents language, video, and robot actions as discrete tokens in one sequence. By choosing which token blocks to mask at inference, the same network can act as a world model (mask goal frames), a robot policy (mask actions), or a video-action model (mask both). The authors report simulation and real-robot results on manipulation benchmarks and closed-loop control.
Why it matters
The paper proposes a unified discrete-token formulation for predicting both future video and action chunks under one masked diffusion objective. It also introduces an action-free co-training mechanism by masking unlabeled action positions and excluding them from loss. The authors show one model can serve multiple roles (world model and policy) and report improved robustness from robot-video pretraining plus human-video co-training.
Method
- Masked diffusion transformer over a shared discrete vocabulary: language, video frames (encoded tokens), and robot action tokens are placed in one sequence and predicted by unmasking masked positions in parallel.
- Action-free co-training: for action-free video, action positions are filled with mask tokens and excluded from the loss; robot labeled demonstrations and human action-free video share one batch and one objective.
- Inference masking selects the mode: mask goal frames for world modeling, mask actions for policy (goal-conditioned), or mask both for video-action generation; decoding uses parallel denoising steps with a cached prefix KV under the block-causal mask.
Limitation
At our scale, co-training yields higher task scores than pretraining alone, however the difference is not statistically separable. More diverse action-free corpora would provide a clearer signal.
Abstract (from arXiv)
Vision-language-action models (VLAs) inherit strong semantic grounding from pretrained vision-language backbones but are typically optimized for predicting actions rather than future observations. They can see and act, but they do not imagine the future before acting. World action models (WAMs) built on video diffusion backbones can imagine but treat language as frozen conditioning on a continuous latent space. Unified models bring these modalities into one architecture, but they either decode autoregressively, one token at a time, or keep video continuous with an auxiliary action head. In this work, we present SUAVE, a Single vocabulary Unified Action-Video modEl in which a masked diffusion transformer generates video and actions conditioned on language, with all three modalities represented as discrete tokens in a shared sequence. Choosing which tokens to mask at inference turns the same network into a world model, a robot policy, or a video-action model. For action-free co-training, the action positions of unlabeled video are filled with mask tokens and excluded from the loss. Simulation and real-world experiments demonstrate two findings. First, a single SUAVE model predicts long-horizon video and acts as a policy, competitive with dedicated world models and specialized action policies on static and dynamic manipulation tasks. On a real robot, our model generates subgoal images and an action chunk spanning one second of motion in 1,030 ms on an RTX 5090 GPU, sustaining closed-loop control at 2.5 actions per second. Second, pretraining on robot video and co-training on human video substantially improves policy performance and zero-shot robustness to distribution shift. Together, these results show that masked diffusion is a practical and versatile foundation for unified video-action modeling.
Related papers
- UniWALM: Unified World Action Language Model for Embodied Robot Control
- UniWAM: Unified World-Action Model
- TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
- Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination
- Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-06 UTC
SUAVE: Unified Video-Action Models via Masked Diffusion
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.04009
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-language-action models (VLAs) inherit strong semantic grounding from pretrained vision-language backbones but are typically optimized for predicting actions rather than future observations. They can see and act, but they do not imagine the future before acting. World action models (WAMs) built on video diffusion backbones can imagine but treat language as frozen conditioning on a continuous latent space. Unified models bring these modalities into one architecture, but they either decode autoregressively, one token at a time, or keep video continuous with an auxiliary action head. In this work, we present SUAVE, a Single vocabulary Unified Action-Video modEl in which a masked diffusion transformer generates video and actions conditioned on language, with all three modalities represented as discrete tokens in a shared sequence. Choosing which tokens to mask at inference turns the same network into a world model, a robot policy, or a video-action model. For action-free co-training, the action positions of unlabeled video are filled with mask tokens and excluded from the loss. Simulation and real-world experiments demonstrate two findings. First, a single SUAVE model predicts long-horizon video and acts as a policy, competitive with dedicated world models and specialized action policies on static and dynamic manipulation tasks. On a real robot, our model generates subgoal images and an action chunk spanning one second of motion in 1,030 ms on an RTX 5090 GPU, sustaining closed-loop control at 2.5 actions per second. Second, pretraining on robot video and co-training on human video substantially improves policy performance and zero-shot robustness to distribution shift. Together, these results show that masked diffusion is a practical and versatile foundation for unified video-action modeling.