Driving · 2026-11-04

HiLo-WAM: Decoupling Long-Horizon Task Evolution and Local Dynamics in World Action Models

Xingyu Wang, Yanjiang Guo, Zhijing Ding, Jiaxin Fang, Zeyu Lai, Jiayu Dong, Weikang Wan, Jianyu Chen, Yuxiang Gao, Mengdi Xu

arXiv:2611.05023PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

HiLo-WAM is a World Action Model for long-horizon robotic manipulation that decouples visual world modeling into two streams: a sparse long-horizon stream for historical task context and coarse future evolution, and a dense local dynamics stream for short-horizon robot–object interactions. It uses a dual-DiT architecture with structured cross-stream attention and progressive inference (partial/joint denoising and dynamic step skipping).

Why it matters

Most WAMs model dense short-horizon dynamics, giving limited long-range task context. HiLo-WAM explicitly separates long-horizon task evolution from local interaction dynamics, enabling action generation to leverage both temporal scales while reducing computation via progressive inference.

Method

  • Decouples visual world modeling into a sparse long-horizon stream (historical context + coarse future evolution) and a dense local dynamics stream (short-horizon robot–object interactions), processed within a dual-DiT architecture.
  • Uses shared Video DiT with structured cross-stream attention to exchange information between visual streams and an Action DiT for action chunk generation.
  • Reduces inference cost with progressive stream-wise partial denoising, joint denoising across streams, and dynamic step skipping based on cosine similarity between consecutive velocity predictions.
Abstract (from arXiv)

World Action Models (WAMs) improve robotic manipulation by jointly modeling visual dynamics and robot actions. However, most existing WAMs model dense short-horizon visual dynamics, capturing local interactions but providing limited long-range task context. Extending prediction horizons increases uncertainty and computation while entangling task-level evolution with fine-grained local dynamics, obscuring their dependencies and distinct roles in action generation. We propose HiLo-WAM, which decouples long-horizon task evolution from local visual dynamics while jointly modeling their interaction with robot actions. A sparse long-horizon visual stream captures historical task context and future evolution, while a dense local dynamics visual stream models short-horizon robot-object interactions. The two streams operate at different temporal scales within a unified dual-DiT architecture and guide actions through structured cross-stream attention. Progressive inference further reduces unnecessary computation through stream-wise partial denoising, joint denoising, and dynamic skipping. Experiments demonstrate strong long-horizon manipulation performance: HiLo-WAM achieves a 93.70% average success rate on RoboTwin 2.0 and a 17.43 progress score on RoboDojo long-horizon tasks, with robust in-domain and OOD performance on the Fourier GR-3 robot. Ablations confirm the complementary contributions of long-horizon task evolution and local interaction dynamics, while temporal-scale analyses highlight the importance of balancing temporal coverage and resolution. Progressive inference further achieves a 2.93x speedup while improving performance.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-03 · Indexed by Pub-AI: 2026-11-04 UTC

preprint

HiLo-WAM: Decoupling Long-Horizon Task Evolution and Local Dynamics in World Action Models

Xingyu Wang, Yanjiang Guo, Zhijing Ding, Jiaxin Fang, Zeyu Lai, Jiayu Dong, Weikang Wan, Jianyu Chen, Yuxiang Gao, Mengdi Xu

Preprint posted: 2026-11-03 · Original record ↗

arxiv
2611.05023
Explicit source versions (1)
  • v1 · 2026-11-03
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-04 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

World Action Models (WAMs) improve robotic manipulation by jointly modeling visual dynamics and robot actions. However, most existing WAMs model dense short-horizon visual dynamics, capturing local interactions but providing limited long-range task context. Extending prediction horizons increases uncertainty and computation while entangling task-level evolution with fine-grained local dynamics, obscuring their dependencies and distinct roles in action generation. We propose HiLo-WAM, which decouples long-horizon task evolution from local visual dynamics while jointly modeling their interaction with robot actions. A sparse long-horizon visual stream captures historical task context and future evolution, while a dense local dynamics visual stream models short-horizon robot-object interactions. The two streams operate at different temporal scales within a unified dual-DiT architecture and guide actions through structured cross-stream attention. Progressive inference further reduces unnecessary computation through stream-wise partial denoising, joint denoising, and dynamic skipping. Experiments demonstrate strong long-horizon manipulation performance: HiLo-WAM achieves a 93.70% average success rate on RoboTwin 2.0 and a 17.43 progress score on RoboDojo long-horizon tasks, with robust in-domain and OOD performance on the Fourier GR-3 robot. Ablations confirm the complementary contributions of long-horizon task evolution and local interaction dynamics, while temporal-scale analyses highlight the importance of balancing temporal coverage and resolution. Progressive inference further achieves a 2.93x speedup while improving performance.