Video World Models · 2026-10-22
LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces
Xinyu Wang, Huafeng Shi, Yue Ma, Zijie Meng, Teng Xi, Yuzhuo Chen, Yucheng Wang, Ruicheng Zhang, Guoxin Zhang, Pengfei Wan, Xiu Li, Fangneng Zhan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
LiveAct is a real-time multi-agent video world model for interactive environments with heterogeneous action interfaces. It introduces a Heterogeneous Action Interface (HAI) that binds each indexed action stream to its intended visual agent (Agent-Aware Spatial Modulation) and adapts action semantics across tasks while preserving synchronized numerical values (Schema-Guided Action Adaptation). A causal generator enables efficient streaming.
Why it matters
The authors study multi-agent video world modeling where multiple controlled agents require consistent joint futures, and where control inputs can mix binary game actions with continuous robot commands. LiveAct’s HAI targets two interface failures: misattribution of which agent a stream controls, and collapse of action meaning across heterogeneous task-specific action spaces. The work reports both agent-attribution accuracy without location/mask supervision and real-time streaming throughput.
Method
- Agent-Aware Spatial Modulation: uses agent descriptions encoded with umT5 to derive language-driven spatial gates and agent-specific attention, encouraging each action stream to affect its corresponding visual agent without inference-time spatial supervision.
- Schema-Guided Action Adaptation: encodes each control dimension’s schema (e.g., attack/steering/robot command) as a semantic token and injects the synchronized numerical value via affine modulation, keeping control meaning decoupled from task-specific vector layouts.
- Streaming causal generation: transfers the full HAI to a blockwise causal generator using teacher forcing for autoregressive tuning and distribution matching distillation for streaming rollouts.
Limitation
Table 3 notes ablations, but the provided text does not state any explicit limitation of the method or evaluation setup beyond what is directly reported.
Abstract (from arXiv)
Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface. We study multi-agent video world modeling, where one model generates a joint visual future for multiple controlled agents while interpreting heterogeneous controls ranging from binary game buttons to continuous robot commands. This task presents two complementary challenges. First, every agent-indexed action stream must control its intended visual entity independently. Second, the model must adapt the meaning of each control variable across different tasks, where the same tensor dimension may denote an attack, steering command, or robot command. We introduce LiveAct, centered on a Heterogeneous Action Interface (HAI) whose two components mirror this decomposition. To support independent control, Agent-Aware Spatial Modulation uses language-driven spatial gates and agent-specific attention to bind each action stream to its intended visual agent. To accommodate heterogeneous controls, Schema-Guided Action Adaptation describes each control variable in language and modulates its semantic token with the synchronized numerical value. We further transfer the complete interface to a blockwise causal generator through teacher forcing and rollout-aware distribution matching, enabling efficient streaming generation while retaining agent-specific control and action semantics. We instantiate the formulation on six multi-agent sources spanning games and multi-robot manipulation, and evaluate visual quality, action following, agent attribution, and latency. Across games and robot manipulation, LiveAct reaches 0.898 and 0.942 agent-attribution accuracy without agent locations or identity crops, and the causal generator attains 17.70 FPS at $512\times 448$.
Related papers
- Agentic World Action Models
- AgentVQA: A Unified Benchmark for Agentic Visual Understanding
- OverSeer: Multi-Agent World Models Beyond What Agents See
- LAW: Unifying Language, Action, and World Modeling for Agentic Physical Intelligence
- Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC
LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.29408
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface. We study multi-agent video world modeling, where one model generates a joint visual future for multiple controlled agents while interpreting heterogeneous controls ranging from binary game buttons to continuous robot commands. This task presents two complementary challenges. First, every agent-indexed action stream must control its intended visual entity independently. Second, the model must adapt the meaning of each control variable across different tasks, where the same tensor dimension may denote an attack, steering command, or robot command. We introduce LiveAct, centered on a Heterogeneous Action Interface (HAI) whose two components mirror this decomposition. To support independent control, Agent-Aware Spatial Modulation uses language-driven spatial gates and agent-specific attention to bind each action stream to its intended visual agent. To accommodate heterogeneous controls, Schema-Guided Action Adaptation describes each control variable in language and modulates its semantic token with the synchronized numerical value. We further transfer the complete interface to a blockwise causal generator through teacher forcing and rollout-aware distribution matching, enabling efficient streaming generation while retaining agent-specific control and action semantics. We instantiate the formulation on six multi-agent sources spanning games and multi-robot manipulation, and evaluate visual quality, action following, agent attribution, and latency. Across games and robot manipulation, LiveAct reaches 0.898 and 0.942 agent-attribution accuracy without agent locations or identity crops, and the causal generator attains 17.70 FPS at $512\times 448$.