Driving · 2026-10-09

Cross-Embodiment Robot Foundation World Models with Latent Actions

Huang Huang, Sriram Yenamandra, Arjun Majumdar, Elie Aljalbout, Tushar Nagarajan, Tsung-Yen Yang, Akshara Rai, Michael Rabbat, Li Fei-Fei, Jiajun Wu, Tingfan Wu, Franziska Meier

arXiv:2610.10846 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce LAC-WM, a robot world model that uses a learned unified latent action space shared across embodiments. Compared to EAC-WM, which conditions on explicit motion labels via embodiment-specific action encoders, LAC-WM improves cross-embodiment downstream performance: up to +46.7% on dexterous manipulation and +11.7% on a modified LIBERO. The unified latent action space also enables downstream scaling with the number of pretraining embodiments.

Why it matters

Cross-embodiment generalization in robot world models is challenged by diverse embodiments and action spaces. The paper argues that explicit action conditioning yields disjoint action representations across embodiments, hurting adaptation to unseen robots. By learning a unified latent action space with latent action conditioning, the authors aim to improve future prediction quality and downstream planning/control performance when transferring to previously unseen embodiments.

Method

Pretrain LAC-WM with an inverse dynamics model (IDM) that encodes pairs of consecutive image observations into latent actions, and a forward dynamics model (FDM) that predicts next-frame visual latents conditioned on those latent actions.
Use a motion decoder trained with an auxiliary motion decoding loss to reconstruct ground-truth motion labels from latent actions, and adopt cross-augmentation to mitigate shortcut solutions.
For unseen embodiments, finetune using an action projector (MLP) that maps raw robot actions into the pretrained unified latent action space, enabling action-conditioned imagined rollouts.

Limitation

The paper does not state limitations in the provided text.

Abstract (from arXiv)

The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-09 UTC

preprint