Driving · 2026-10-02

SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation

Juyi Sheng, Hua Wang, Mengyuan Liu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SkeleWAM is a compact world action model for robotic manipulation that represents each scene as a sparse 3D skeleton of robot joints, object centers, and interaction points built online from RGB-D and proprioception. It jointly learns action generation and future skeleton prediction with flow matching; the future branch is omitted at inference. On LIBERO-Plus it reports 85.9% overall success with 57.1M parameters, outperforming Cosmos-Policy by 3.7 points, and reports robustness under camera perturbations.

Why it matters

The authors argue that sparse, explicit robot–object geometry avoids relying on video/latent representations that capture interaction geometry only implicitly. By using future skeleton prediction as auxiliary supervision during training, SkeleWAM aims to improve world-action learning without requiring visual reconstruction and while keeping the model compact. Reported gains on LIBERO-Plus and additional results on real robot tasks support the effectiveness of the skeleton state space for robust, parameter-efficient manipulation.

Method

Represent manipulation scenes as a sparse 3D skeleton (robot joints/end-effector positions plus object centers and interaction points) constructed online from RGB-D and robot proprioception, and encode it as geometric tokens.
Jointly train action generation and future skeleton prediction (auxiliary geometric supervision) using flow matching; omit future-skeleton prediction at inference.
Use Medoid Action Consensus (MAC) to select among stochastic sampled action trajectories without a reward/value model.

Limitation

Layout success under camera and robot initial-state perturbations is limited: 'Performance under layout perturbations remains limited at 66.6%.' (and 'Layout success reaches only 69.0% even with simulator-derived coordinates.')

Abstract (from arXiv)

World action models (WAMs) combine robot action generation with future state prediction. Existing WAMs typically predict videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control. We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of robot joints, object centers, and interaction points. Constructed online from current RGB-D observations and robot proprioception, the skeleton provides a unified geometric state for action generation and future skeleton prediction. Future skeleton prediction provides additional geometric supervision for action learning without requiring visual reconstruction. At inference, SkeleWAM generates actions directly from the current skeleton and language instruction, while Medoid Action Consensus (MAC) serves as an auxiliary consensus strategy for stochastic action samples. On LIBERO-Plus, SkeleWAM achieves an overall success rate of 85.9% with 57.1M parameters, outperforming Cosmos-Policy by 3.7 percentage points. These results demonstrate that sparse 3D robot--object structure provides an effective state space for robust and parameter-efficient world action learning. The project is available at https://skelewam-project.github.io/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC

preprint