Video World Models · 2026-10-26
UniAction: Unified Action Conditioning for World Models
Bowen Xue, Zizhang Li, Hong-Xing Yu, Jiajun Wu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
UniAction unifies action conditioning for world models by representing any action as a sparse set of action elements with (1) discrete semantic identities and (2) continuous spatiotemporal grounding coordinates. These coordinates are mapped into the pretrained video model’s native positional geometry, and cardinality-calibrated attention compensates for varying numbers of action elements. On four action forms (force, camera control, hand pose, robot pose), UniAction outperforms conditioning alternatives and a jointly trained single model surpasses action-specific experts.
Why it matters
Action-annotated data for world models is scarce and heterogeneous actions are typically handled by separate models or action-specific conditioning mechanisms. UniAction provides a shared conditioning interface so different action forms can condition a single video world model through the same mechanism, mitigating data fragmentation and enabling joint learning across heterogeneous action data.
Method
- Represents any action as sparse action elements with discrete semantic identity (token indexed from a shared codebook) and continuous spatiotemporal grounding coordinate p=(t,y,x).
- Maps action coordinates into the pretrained video backbone’s native positional geometry so action and video tokens share the same positional relations and self-attention mechanism.
- Uses cardinality-calibrated attention to remove bias from differing action-token counts when computing attention between video queries and action keys.
Abstract (from arXiv)
Actions that condition world models are highly heterogeneous, and thus existing action-conditioned world models are typically built for individual action types, fragmenting already limited action-annotated data across separate models. We introduce UniAction, a unified action-conditioning framework that represents any action as a sparse set of action elements, each with a discrete semantic identity and a continuous spatiotemporal grounding coordinate. By mapping these coordinates directly into the pretrained video model's native positional space, UniAction allows heterogeneous actions to condition a shared world model within the same spatiotemporal structure. We further introduce cardinality-calibrated attention to address the bias from differing action representation granularities. We evaluate UniAction on four representative action forms: force, camera control, hand pose, and robot pose. UniAction outperforms conditioning alternatives in action following, and a single jointly trained model surpasses the action-specific experts, showing that heterogeneous action data benefit from unified model and joint training.
Related papers
- UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- UniWAM: Unified World-Action Model
- Can World and Action Modeling Benefit Each Other under Unified Causal Training?
- SUAVE: Unified Video-Action Models via Masked Diffusion
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-23 · Indexed by Pub-AI: 2026-10-26 UTC
UniAction: Unified Action Conditioning for World Models
Preprint posted: 2026-10-23 · Original record ↗
- arxiv
- 2610.32977
Explicit source versions (1)
- v1 · 2026-10-23
- Preprint available
Source date: 2026-10-23 · First detected: 2026-10-27 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · atom:published (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Actions that condition world models are highly heterogeneous, and thus existing action-conditioned world models are typically built for individual action types, fragmenting already limited action-annotated data across separate models. We introduce UniAction, a unified action-conditioning framework that represents any action as a sparse set of action elements, each with a discrete semantic identity and a continuous spatiotemporal grounding coordinate. By mapping these coordinates directly into the pretrained video model's native positional space, UniAction allows heterogeneous actions to condition a shared world model within the same spatiotemporal structure. We further introduce cardinality-calibrated attention to address the bias from differing action representation granularities. We evaluate UniAction on four representative action forms: force, camera control, hand pose, and robot pose. UniAction outperforms conditioning alternatives in action following, and a single jointly trained model surpasses the action-specific experts, showing that heterogeneous action data benefit from unified model and joint training.
