Robotics · 2026-10-05

World Action Learning via Interaction-Centric Spectral Latent Guidance

Zhiming Liu, Yikun Miao, Ying Chen, Hongrui Yin, Fangqi Zhu, Xiaoyi Pang, Quanxin Shou, Zhengyang Yan, Haodong Wang, Song Guo

arXiv:2610.03607 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

WING transfers interaction knowledge from egocentric videos to robot policies by learning interaction-centric latent actions while suppressing observer motion, then extracting slowly varying low-frequency components in the spectral domain as guidance for action generation. The authors report average success rates of 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1, plus results on four real-world manipulation tasks under diverse generalization settings.

Why it matters

The authors target cross-embodiment robot learning from egocentric video without direct action supervision, addressing two issues in latent-action transfer: reconstruction-based latent actions can entangle nuisance observer motion (e.g., ego-camera motion), and human/robot temporal dynamics can differ. WING’s motion decoupling and low-frequency spectral guidance are presented as scalable mechanisms for extracting transferable interaction structure for robot control.

Method

WING-LAM learns interaction-centric latent actions by decomposing frame transitions into global observer-induced motion and local interaction motion, training an interaction latent that reconstructs interaction targets while suppressing observer motion.
Spectral Latent Guidance applies DCT to latent action trajectories and retains the lowest K low-frequency components as transferable guidance, conditioning robot action generation at inference.
A lightweight predictor estimates the low-frequency spectral guidance from current observation, instruction, and proprioception (when available), and the action model attends to these guidance tokens during action generation.

Limitation

The authors report that “inference latency remains a limitation of the current implementation” due to additional computation introduced by the dense world-modeling architecture.

Abstract (from arXiv)

Learning general-purpose robot policies requires large-scale real-world interaction data, yet collecting robot demonstrations remains expensive and difficult to scale. Egocentric videos offer abundant human interaction experience with task-relevant semantics for robotic manipulation, but direct transfer is challenging for two reasons: latent actions inferred from frame reconstruction can be dominated by nuisance variation such as ego-camera motion, and human and robot behaviors often exhibit different temporal dynamics. We propose WING (World Action Learning via INteraction-Centric Spectral Latent Guidance), a framework for transferring interaction knowledge from egocentric videos to robot policies. WING first separates observer-induced motion from hand-object interaction and distills the interaction-centric component into latent actions. It then exploits the observation that cross-embodiment task semantics are concentrated in slowly varying temporal structures, identifying shared low-frequency components between egocentric latent actions and robot behaviors in the spectral domain and using them to guide action generation. WING achieves average success rates of 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa-GR1, and also performs strongly across four real-world manipulation tasks under diverse generalization settings. These results show that interaction-centric spectral guidance provides an effective and scalable way to transfer physical interaction knowledge from human egocentric video to robot control. Project page: https://mikuz12.github.io/wing/

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC

preprint