Robotics · 2026-11-05

ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

Cheng-Chun Hsu, Peter Stone

arXiv:2611.05571PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

ELA is an end-to-end framework for learning robot imitation from video demonstrations using latent actions. It jointly trains a Latent Action Model (inverse + forward dynamics), a latent action policy, and a continuous action decoder, optimizing latent actions for both policy learning and action decoding. The authors report consistent improvements over latent-action baselines on simulated and real-world manipulation tasks.

Why it matters

The authors address two issues in prior latent-action methods: latent actions learned in a two-stage way may be hard to model from observations and hard to decode into continuous control. ELA ties latent-action learning to downstream control by jointly optimizing latent actions for policy learning and action decoding, and claims integration with existing imitation learning via a lightweight action decoder.

Method

  • ELA trains a latent action model with an inverse dynamics model (IDM) to infer latent actions from observation pairs and a forward dynamics model (FDM) for future frame reconstruction.
  • It factorizes the policy as a latent action policy (predicts latent actions from single observations) plus an action decoder (maps latent actions to continuous robot actions), trained with an action reconstruction objective using action-labeled data.
  • It uses representation alignment with a frozen perceptual encoder (DINOv3) and alternating update passes (LAM-update vs policy-update) to prevent latent action collapse during joint training.

Limitation

The only exception noted is the open-oven task, where Ela_D​P does not provide further gains; the authors state the task demands highly precise continuous control under strong articulation constraints that exceeds the modeling capacity of the IDM when learning from typical human demonstration videos.

Abstract (from arXiv)

Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-05 · Indexed by Pub-AI: 2026-11-05 UTC

preprint

ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

Cheng-Chun Hsu, Peter Stone

First announced by arXiv: 2026-11-05 · Original record ↗

arxiv
2611.05571
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-05 · First detected: 2026-11-05 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.05571v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/