Driving · 2026-11-10

SWAM: Scaling World Action Models with Cross-Modal Alignment

Yu Zhou, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen, Bowen Liu, Jianhan Ma, Weicheng Duan, Jun Lv, Cewu Lu, Yanwei Li

arXiv:2611.12870PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SWAM trains a world action model to jointly predict language-conditioned video changes and robot actions. The authors curate a ~47k-hour multisource dataset and study supervision and architecture choices for preserving language–video alignment during action learning and improving language–video–action control generalization. The final recipe post-trained on RoboTwin Clean demonstrations only reports 88.10%/61.56% success on Easy/Hard and 40.00% mean SR across three real-world tasks, versus 23.33% for the strongest baseline.

Why it matters

The authors address a core issue in world action models: adding action learning can impair pretrained language–video alignment. SWAM provides empirical guidance on data mixture and supervision design (language–video vs language–video–action) and a model architecture that supports joint learning while reusing language–video correspondences. The reported gains on unseen conditions aim to show better control generalization from scaled multisource pretraining and an end-to-end joint prediction objective.

Method

  • Curate a ~47,000-hour multisource pretraining pool (general videos, egocentric videos, UMI demonstrations, robot demonstrations) and use language–video supervision plus temporally aligned action supervision when available.
  • Train an end-to-end world action model with a video DiT branch (initialized from Wan) and an action DiT branch connected through shared masked attention with unidirectional interaction from video to action; action branch has no direct text cross-attention.
  • Use a joint flow-matching objective for language-conditioned video prediction and (only for samples with valid annotations) action prediction from language, video latents, and current state.

Limitation

The paper text does not state a specific limitation.

Abstract (from arXiv)

World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-09 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

SWAM: Scaling World Action Models with Cross-Modal Alignment

Yu Zhou, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen, Bowen Liu, Jianhan Ma, Weicheng Duan, Jun Lv, Cewu Lu, Yanwei Li

Preprint posted: 2026-11-09 · Original record ↗

arxiv
2611.12870
Explicit source versions (1)
  • v1 · 2026-11-09
  1. Preprint available

    Source date: 2026-11-09 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.