Robotics · 2026-10-23

Skeleton-Grounded Latent Action Model for Robot World Models

Qi Lv, Jiayi Lin, Yuxin Jiang, Jianzong Wang, Xiang Deng, Mike Shou Zheng

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SkeLAM is a skeleton-guided latent action model for controllable robot world modeling. It decomposes latent actions into a skeleton-grounded kinematic component and a residual component, trained with cross-modal swap reconstruction and branch dropout to support action-conditioned inference. A lightweight action adapter maps robot actions to the kinematic space, followed by world-model fine-tuning. On DROID, it improves end-effector motion and visual fidelity versus DreamDojo.

Why it matters

The authors address a key problem with reconstruction-driven latent actions: they can entangle robot motion with object responses and other scene changes, making it hard to map robot controls to the world model. By separating motion conditioning (kinematics) from complementary dynamics (residual) using skeleton supervision and training tricks (swap reconstruction, branch dropout), SkeLAM aims to produce a more controllable and action-aligned latent action representation for robot simulation.

Method

Decompose RGB-derived latent actions into z_kin (grounded by robot skeleton supervision) and z_res (retaining complementary visual dynamics).
Use cross-modal swap reconstruction: replace the RGB kinematic latent with the paired skeleton kinematic latent while keeping the RGB residual during decoding.
Train an action adapter to map robot actions to the kinematic latent; at inference, generate future observations from the current observation plus robot actions with residual zeroed via branch dropout.

Limitation

The paper states that human-to-robot transfer examples assess visible motion correspondence, not task completion or reliable transfer across arbitrary viewpoints and embodiments.

Abstract (from arXiv)

Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC

preprint