Driving · 2026-11-10
SWAM: Scaling World Action Models with Cross-Modal Alignment
Yu Zhou, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen, Bowen Liu, Jianhan Ma, Weicheng Duan, Jun Lv, Cewu Lu, Yanwei Li
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SWAM trains a world action model to jointly predict language-conditioned video changes and robot actions. The authors curate a ~47k-hour multisource dataset and study supervision and architecture choices for preserving language–video alignment during action learning and improving language–video–action control generalization. The final recipe post-trained on RoboTwin Clean demonstrations only reports 88.10%/61.56% success on Easy/Hard and 40.00% mean SR across three real-world tasks, versus 23.33% for the strongest baseline.
Why it matters
The authors address a core issue in world action models: adding action learning can impair pretrained language–video alignment. SWAM provides empirical guidance on data mixture and supervision design (language–video vs language–video–action) and a model architecture that supports joint learning while reusing language–video correspondences. The reported gains on unseen conditions aim to show better control generalization from scaled multisource pretraining and an end-to-end joint prediction objective.
Method
- Curate a ~47,000-hour multisource pretraining pool (general videos, egocentric videos, UMI demonstrations, robot demonstrations) and use language–video supervision plus temporally aligned action supervision when available.
- Train an end-to-end world action model with a video DiT branch (initialized from Wan) and an action DiT branch connected through shared masked attention with unidirectional interaction from video to action; action branch has no direct text cross-attention.
- Use a joint flow-matching objective for language-conditioned video prediction and (only for samples with valid annotations) action prediction from language, video latents, and current state.
Limitation
The paper text does not state a specific limitation.
Abstract (from arXiv)
World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.
Related papers
- UniWALM: Unified World Action Language Model for Embodied Robot Control
- UniWAM: Unified World-Action Model
- WorldBridge: Bridging Video and Action for Task Generalization in World Action Models
- SUAVE: Unified Video-Action Models via Masked Diffusion
- OpenWAM: An Open Framework for Composable World-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-09 · Indexed by Pub-AI: 2026-11-10 UTC
SWAM: Scaling World Action Models with Cross-Modal Alignment
Preprint posted: 2026-11-09 · Original record ↗
- arxiv
- 2611.12870
Explicit source versions (1)
- v1 · 2026-11-09
- Preprint available
Source date: 2026-11-09 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.12870v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.
