Robotics · 2026-10-21
UniWALM: Unified World Action Language Model for Embodied Robot Control
Rahul Sajnani, Diego Romeres, Mingyuan Zhang, Kai Gao, Carl Qi, Baichuan Huang, Srinath Sridhar, Dylan Randle, Lillian J. Ratliff, Jose Barreiros, Manikantan Nambi, Jeroen Vanbaar
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
UniWALM is a unified world-action-language model that learns video, action, and language in a single shared mixture-of-transformers representation, co-denoising per-modality diffusion heads synchronously and re-consuming generated language as next-step prompt (language-as-memory loop) with no separate planning stage. It also adds Flow Matching with Correction (FMC) to improve action reliability. The authors report improved manipulation and memory-dependent behavior in simulation and real-world tasks.
Why it matters
The paper targets long-horizon embodied control where policies need both action correctness and task memory. UniWALM’s key idea is generating and re-consuming language in-loop as working memory while co-modeling video and action in one synchronous diffusion process, aiming to remove external planners/VLMs/memory banks. The authors claim strong gains on memory-dependent benchmarks and a large lift on a real-world find-the-item task when using textual memory.
Method
- UniWALM (WALM): a single mixture-of-transformers backbone with per-modality diffusion heads that co-denoise future video, actions, and a language stream emitting (subtask || textual memory), which is re-consumed next step as prompt.
- Memory mechanisms inside the shared representation: textual memory re-consumed across control steps; optional visual memory via anchor frame(s); a task-completion channel β that gates when textual memory is updated at subtask boundaries.
- Flow Matching with Correction (FMC): during training, perturb action trajectories and supervise corrective velocity so the denoising process self-corrects without changing inference-time cost.
Limitation
The authors list four main failure modes: (i) boundary misfires where β fires one step late/early, (ii) incorrect subtask prediction, (iii) grasp precision issues on complex-geometry objects and cluttered scenes, and (iv) inference speed when generating a 32-step action chunk with the language expert at 10 denoising steps.
Abstract (from arXiv)
Motivated by the human ability to integrate experience across language, perception, and action, we introduce UniWALM, a unified world-action-language model that learns video, action, and language in a single shared representation, where each modality is both a conditioning signal and a supervision target. We ask: can jointly generating language enrich this shared representation, improving manipulation and supporting memory-dependent behavior? Unlike prior approaches that attach a reasoning or vision-language model to a video-action backbone and run them as distinct stages, UniWALM unifies all three modalities inside a single mixture-of-transformers backbone whose per-modality diffusion heads co-denoise synchronously, decoding subtask and textual memory in the same pass as action with no separate planning stage; the generated language is re-consumed on the next step, closing a language-as-memory loop without an external planner or memory bank. To keep the action stream reliable under this richer objective, we further introduce Flow Matching with Correction (FMC), a training objective that teaches recovery from perturbed action trajectories at no inference-time cost. UniWALM consistently improves over same-backbone baselines (e.g., FastWAM) on compositional pick-and-place (MolmoSpaces) and across the memory-dependent subset of RMBench, where it reaches 34.5% without any pretrained language or reasoning model and exceeds Mem-0's keyframe-equipped setup (28.5%) using only a single anchor frame. On a real-world Find-the-Item task, generating textual memory lifts success from 11.1% to 61.1%, while state-of-the-art VLAs stay at or below chance (<=33.3%).
Related papers
- UniWAM: Unified World-Action Model
- LAW: Unifying Language, Action, and World Modeling for Agentic Physical Intelligence
- SUAVE: Unified Video-Action Models via Masked Diffusion
- EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model -- From Semantic Understanding through Visual Foresight to Action
- Language-Forcing: Advancing Humanoid Loco-Manipulation via Semantic Alignment of Whole-Body World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
UniWALM: Unified World Action Language Model for Embodied Robot Control
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28626
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Motivated by the human ability to integrate experience across language, perception, and action, we introduce UniWALM, a unified world-action-language model that learns video, action, and language in a single shared representation, where each modality is both a conditioning signal and a supervision target. We ask: can jointly generating language enrich this shared representation, improving manipulation and supporting memory-dependent behavior? Unlike prior approaches that attach a reasoning or vision-language model to a video-action backbone and run them as distinct stages, UniWALM unifies all three modalities inside a single mixture-of-transformers backbone whose per-modality diffusion heads co-denoise synchronously, decoding subtask and textual memory in the same pass as action with no separate planning stage; the generated language is re-consumed on the next step, closing a language-as-memory loop without an external planner or memory bank. To keep the action stream reliable under this richer objective, we further introduce Flow Matching with Correction (FMC), a training objective that teaches recovery from perturbed action trajectories at no inference-time cost. UniWALM consistently improves over same-backbone baselines (e.g., FastWAM) on compositional pick-and-place (MolmoSpaces) and across the memory-dependent subset of RMBench, where it reaches 34.5% without any pretrained language or reasoning model and exceeds Mem-0's keyframe-equipped setup (28.5%) using only a single anchor frame. On a real-world Find-the-Item task, generating textual memory lifts success from 11.1% to 61.1%, while state-of-the-art VLAs stay at or below chance (<=33.3%).