Driving · 2026-10-08

RealtimeWAM: How Fast Can I Run My World Action Model?

Huanan Liu, Ye Li, Kangye Ji, Xiaoyu Chen, Hanyun Cui, Yutian Shen, Yuan Meng, Chenglei Wu, Jingyan Jiang, Bo Li, Zhi Wang

arXiv:2610.10079 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

RealtimeWAM is a training-free framework for world action models that reduces inference latency for responsive robot control. It overlaps observation processing with prediction using layerwise dependencies, reduces Prefill contention via hardware-aware token reuse guided by observation change and latency plateaus, and allocates refinement using predicted motion. Evaluated on FastWAM and OpenWAM across RoboTwin, LIBERO, and LIBERO-Plus, it reports large latency reductions with near-native success.

Why it matters

The authors report that WAMs’ high inference cost limits real-time responsive control. RealtimeWAM targets low-latency closed-loop execution without additional training, coordinating parallel execution and adaptive computation to match changing computational demands across a rollout. The paper reports measured latency reductions and small success-rate drops relative to native inference, plus larger gains on real-world tasks.

Method

Dependency-aware parallel schedule: overlap Prefill (observation processing) with action prediction by releasing observation K/V layerwise so prediction can proceed while Prefill continues (including future-video branch when present).
Hardware-aware token reuse: refresh selected observation tokens and reuse cached FFN outputs for others, using offline profiling latency plateaus and ranking tokens by feature change to fill spare capacity.
Motion-adaptive refinement: cache Transformer residuals and allocate additional Transformer refresh evaluations based on a coarse action’s predicted displacement (motion gate), using fewer evaluations for large predicted movements.

Abstract (from arXiv)

World Action Models (WAMs) combine visual dynamics modeling with action generation, but their high inference latency limits responsive robot control. Recent efforts accelerate inference by removing explicit future-video generation at test time, as in FastWAM, an approach that requires a specially tailored architectural design. More general caching strategies exploit feature redundancy, but redundancy alone does not capture the changing computational demands of closed-loop control. To address these challenges, we present RealtimeWAM, a general, training-free framework that coordinates parallel execution with adaptive computation for low-latency inference across diverse WAM architectures. We exploit layerwise dependencies to overlap observation processing with prediction. However, concurrent branches still compete for GPU resources, limiting the benefit of parallel execution. We therefore adapt computation throughout the pipeline through selective reuse, caching observation features in visually stable regions and reusing Transformer residuals while reserving additional refinement for small predicted adjustments. We evaluate RealtimeWAM on FastWAM and OpenWAM across RoboTwin, LIBERO, and LIBERO-Plus. On an RTX 4090, measured mean inference latencies are 24.09 and 63.09 ms, corresponding to average speedups of 8.90$\times$ and 10.67$\times$. Average success rates are 82.75% and 87.41%, respectively, within 0.02 and 0.53 percentage points of native inference. Across five real-world tasks, RealtimeWAM improves average success rates over native inference by 17.2 and 37.2 percentage points on FastWAM and OpenWAM, respectively.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC

preprint