Latent Dynamics · 2026-10-12

LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

Zhiyuan Xiang, Peidong Liu, Mingyang Li, Jiawei Li, Wenhow Li, Wenbo Ding, Xiang Deng

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

LAPT-Scale pretrains embodied VLA policies using human egocentric videos without robot action labels. It introduces LAPT-Scale-LAM, a continuous latent action model trained to explain frame-to-frame transitions using perceptual, semantic (DINOv2), and optical-flow (RAFT) alignment. LAPT-Scale-Policy uses these latent actions for Stage-1 pre-training, then performs target-robot embodiment alignment and task post-training. The authors report improved real-robot success and scaling gains with more annotated human video.

Why it matters

Robotic generalist policies are limited by costly action-labeled interaction data. The authors propose a way to turn large-scale human videos into scalable action-like supervision via continuous latent actions, then transfer that supervision to robot policies through embodiment alignment. They report both simulation and real-robot gains, including that increasing annotated human-video hours improves real-robot success and that the learned latent annotations generalize to unseen visual shifts.

Method

LAPT-Scale-LAM learns continuous latent actions by encoding transitions between adjacent frames and training a spatio-temporal VAE to reconstruct the next frame while adding perceptual (LPIPS), semantic (DINOv2), and optical-flow (RAFT) alignment terms.
LAPT-Scale-Policy trains a VLA policy in three stages: human-video latent-action pre-training on LAM-produced latent actions, target-robot embodiment alignment on robot data, and downstream task-specific post-training.
The authors use a continuous latent space (vs prior vector-quantized LAMs) to preserve fine-grained motion and interaction dynamics, and they train with multiple alignment signals beyond pixel reconstruction.

Limitation

The main limiting assumption is that visual latent actions learned from human videos can become useful robot-policy supervision after embodiment alignment; they are not guaranteed to align with independently controllable robot degrees of freedom. The approach may also fail when videos contain weak manipulation signal, large embodiment gaps, heavy occlusion, or visual changes dominated by ego-camera motion.

Abstract (from arXiv)

Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pre-training paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and optical-flow alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-robot success from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC

preprint