Latent Dynamics · 2026-10-16

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

Meng Cao, Can Zhao, Jinjie Wei, Haonan Zhao, Yulong Zhang, Duo Han, Xiaofan Li, Ziwu Song, Rongtao Xu, Ruyi Gan, Hao Wang, Qian Wang, Daolin Ma, Ian Reid, Xiaodan Liang

arXiv:2610.20904 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

EgoJEPA is a three-stage action-conditioned world modeling framework for dexterous manipulation from egocentric human videos. It avoids rule-based human-to-robot retargeting by learning embodiment-agnostic, effect-aware latent actions via JEPA-style prediction, then trains a latent-action world model, and finally adapts to different robots with lightweight action adapters. On five real-world tasks across three embodiments, it reports 74% average success under out-of-domain evaluation vs 65% for a retargeting baseline.

Why it matters

The authors target failures of retargeting pipelines that require geometric recalibration across embodiments and can propagate hand-tracking errors into robot control. EgoJEPA’s learned latent-action interface aims to decouple human-video priors from embodiment-specific control parameterizations, improving cross-embodiment scaling and robustness to action-annotation noise.

Method

Stage 1: learn effect-aware latent actions from egocentric human videos using JEPA-style visual prediction conditioned on adjacent-frame hand-motion differences (human action = annotation differences).
Stage 2: train a latent-action world model that jointly predicts future visual dynamics and latent actions via joint flow matching, supervised by frozen action embeddings rather than retargeted robot commands.
Stage 3: ground the shared latent action space into a target robot embodiment using lightweight input/output action adapters, expressing robot actions as temporal differences and adapting only Stage 3 per robot.

Limitation

Performance drops on out-of-domain tasks requiring fine-grained contact dynamics (e.g., bottle cap unscrewing). The authors also note that extreme degradation in egocentric video actions (e.g., severe temporal misalignments or prolonged occlusions) can still distort latent representations and bottleneck downstream task performance.

Abstract (from arXiv)

Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-16 UTC

preprint