Latent Dynamics · 2026-11-13

SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

Jiachun Jin, Zetong Zhou, Haonan Wang, Xiao Yang, Jun Zhu, Zhijie Deng

arXiv:2611.17110PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SALA trains a latent action tokenizer with semantic alignment across human and robot embodiments using unpaired atomic-action language descriptions. Language similarities form soft neighborhoods that move human latents toward control-grounded robot latents (robot latents anchored by action reconstruction). The aligned shared latent action space enables scaling: more general human-video pre-training reduces robot action prediction error; more task-specific human-video post-training increases real-robot success rates.

Why it matters

Embodied foundation models are limited by costly robot interaction data and by latent action models that may cluster by visual appearance instead of action semantics. SALA uses language as relational supervision from unpaired human/robot clips to align latent actions across embodiments, turning human egocentric videos into actionable supervision for the same robot action expert and enabling dual-axis scaling.

Method

  • Ground robot latents in control by training a robot-only action decoder to reconstruct robot action chunks from tokenized latent transitions; this anchors robot latents for alignment.
  • Use unpaired atomic-action language descriptions to compute language similarities between human and robot clips, convert them into soft neighborhood structures, and supervise cross-embodiment latent neighborhood correspondence (aligned by moving human latents toward robot anchors).
  • Freeze the aligned tokenizer after training and use it to provide latent-action supervision (token distributions) for an embodied foundation policy on both human and robot videos; human data trains latent-action prediction without action labels.

Limitation

The paper does not state a specific limitation in the provided text.

Abstract (from arXiv)

Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-12 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

Jiachun Jin, Zetong Zhou, Haonan Wang, Xiao Yang, Jun Zhu, Zhijie Deng

Preprint posted: 2026-11-12 · Original record ↗

arxiv
2611.17110
Explicit source versions (1)
  • v1 · 2026-11-12
  1. Preprint available

    Source date: 2026-11-12 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.