Video World Models · 2026-10-20
Language-Forcing: Advancing Humanoid Loco-Manipulation via Semantic Alignment of Whole-Body World Action Models
Yanjun Li, Jiarui Ye, Weiyu Guo, Zixiao He, Kangjie Ye, Dihan Wang, Yishuo Cheng, Zishang Xiang, Mingju Gao, Hao Tang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Language-Forcing is a training strategy for whole-body World Action Models (WAMs) that uses language supervision to emphasize task-relevant transitions in intermediate future-visual features. A frozen vision-language model generates text targets from paired current/future frames and task context; a frozen text encoder aligns these embeddings with intermediate action-conditioned future-visual features, while retaining the original visual and action objectives. Auxiliary components are training-only (no inference overhead).
Why it matters
The authors target a limitation of visual prediction losses in whole-body control: they may overemphasize incidental scene changes (e.g., viewpoint shifts) and underweight subtle but task-critical events (e.g., first contact). By aligning future-visual features with language descriptions of task progress and likely next states, the method aims to make the learned representation more relevant to action decisions.
Method
- Train a whole-body WAM with an added language loss: a frozen vision-language teacher outputs descriptions of task progress and future states from paired current/future frames plus task context.
- Encode teacher text with a frozen UMT5 encoder, project intermediate action-conditioned future-visual features, and align them using a cosine loss between projected visual features and language embeddings.
- Use auxiliary language components only during training; at inference the language-alignment branch is omitted, preserving the action expert’s deployment path.
Abstract (from arXiv)
World Action Models (WAMs) jointly predict robot actions and future observations. However, their visual prediction objectives do not explicitly distinguish task-relevant transitions from incidental scene changes. For whole-body policies, this distinction matters because task progress depends on coordinated changes in body pose, contact, and object state. We introduce LANGUAGE-FORCING, a training strategy that uses language to emphasize task-relevant changes in the future visual representations of a whole-body WAM. Given paired current and ground- truth short-horizon future frames, a frozen vision-language model assesses task progress and predicts plausible subsequent state changes. We encode these descriptions with a frozen text encoder and align their embeddings with intermediate, action-conditioned future-visual features, complementing the original visual and action prediction objectives. All auxiliary components are used only during training, introducing no additional inference overhead. On HumanoidArena, LANGUAGE-FORCING achieves an average success rate of 79.17%, exceeding the strongest competing policy by 9.46 percentage points and the same WAM without language alignment by 10.77 percentage points. Across five real-world tasks, language alignment raises average success from 53.33% to 71.33% over the same WAM baseline.
Related papers
- Being-M0.7: A Latent World-Action Model for Humanoid Robots
- UniWAM: Unified World-Action Model
- UniWALM: Unified World Action Language Model for Embodied Robot Control
- Vision-Language-Delta-Action: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation
- Humanoid World Action Model With Joint State--Action Generation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-28 · Indexed by Pub-AI: 2026-10-20 UTC
Language-Forcing: Advancing Humanoid Loco-Manipulation via Semantic Alignment of Whole-Body World Action Models
Preprint posted: 2026-09-28 · Original record ↗
- arxiv
- 2610.23894
Explicit source versions (1)
- v1 · 2026-09-28
- Preprint available
Source date: 2026-09-28 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World Action Models (WAMs) jointly predict robot actions and future observations. However, their visual prediction objectives do not explicitly distinguish task-relevant transitions from incidental scene changes. For whole-body policies, this distinction matters because task progress depends on coordinated changes in body pose, contact, and object state. We introduce LANGUAGE-FORCING, a training strategy that uses language to emphasize task-relevant changes in the future visual representations of a whole-body WAM. Given paired current and ground- truth short-horizon future frames, a frozen vision-language model assesses task progress and predicts plausible subsequent state changes. We encode these descriptions with a frozen text encoder and align their embeddings with intermediate, action-conditioned future-visual features, complementing the original visual and action prediction objectives. All auxiliary components are used only during training, introducing no additional inference overhead. On HumanoidArena, LANGUAGE-FORCING achieves an average success rate of 79.17%, exceeding the strongest competing policy by 9.46 percentage points and the same WAM without language alignment by 10.77 percentage points. Across five real-world tasks, language alignment raises average success from 53.33% to 71.33% over the same WAM baseline.