Robotics · 2026-11-13
EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming
Yaru Niu, Binghong Chen, Changwei Yao, Shangtao Li, Zhenlong Fang, Revanth Krishna Senthilkumaran, Shuai Zhou, Yuemin Mao, Kalie Ching, Po-Ju Chao, Yudong Luo, Peiqi Yu, Heng Zhang, Hao Zhang, Bingqing Chen, Chen Qiu, H. Eric Tseng, Ye Yuan, Umar Iqbal, Changliu Liu, Jonathan Francis, Ding Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
EgoX is an egocentric cross-embodiment manipulation framework that pretrains a Modularized Cross-Embodiment Transformer (MXT+) with a training-only Embodiment Dreaming objective on mixed demonstrations from multiple humans and robots, then fine-tunes on a target embodiment. The authors report higher average normalized task progress (0.63→0.80) and larger gains under out-of-distribution (0.53→0.77) across 13 task–embodiment pairs.
Why it matters
Training on combined demonstrations across different robot embodiments is difficult due to morphology, sensing, and control interface differences. EgoX aims to transfer experience via semantically corresponding observation/action modalities while keeping sensing and control embodiment-specific, using MXT+ modular interfaces and an embedding-space prediction objective to encourage shared dynamics modeling.
Method
- EgoX organizes heterogeneous human/robot/sim demonstrations into a shared modality-structured egocentric observation-action representation (unified egocentric frames; end-effector keypoints described geometrically).
- MXT+ uses embodiment-specific tokenizers for each observation modality and embodiment-specific action experts, connected through a shared Transformer encoder-decoder trunk across embodiments.
- Embodiment dreaming is a training-only auxiliary objective that predicts future embodiment latents from current shared representations to encourage embodiment-aware transition dynamics modeling.
Limitation
The provided text does not state explicit limitations.
Abstract (from arXiv)
Generalist manipulation policies can benefit from the complementary experience of humans, real robots, and simulated agents. However, differences in morphology, sensing, and control interfaces make it difficult to learn effectively from their combined demonstrations. We introduce EgoX, an egocentric cross-embodiment learning framework that organizes heterogeneous demonstrations through semantically corresponding observation and action modalities. EgoX pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on this mixture and fine-tunes it on target-embodiment demonstrations. Within each embodiment, MXT+ assigns a dedicated tokenizer to each observation modality and a dedicated expert to each action modality, with a Transformer trunk shared across embodiments. A training-only embodiment-dreaming objective predicts latent representations of future visual and proprioceptive states, encouraging the shared trunk to model each embodiment's dynamics. Across 13 task-embodiment pairs spanning five tasks and six robot embodiments, including five physical robots and one simulated humanoid, cross-embodiment pretraining raises MXT+'s average normalized task progress from 0.63 to 0.80 over training from scratch. The gains are particularly pronounced under out-of-distribution conditions, where average task progress increases from 0.53 to 0.77. Ablations demonstrate the benefits of broader pretraining mixtures, modular interfaces, and embodiment dreaming, with dreaming producing more similar latent geometry across embodiments. Project page: egox-dream.github.io.
Related papers
- EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
- Cross-Embodiment Motion Planning by Coupling Body-Agnostic World Model with Differentiable Ego-Dynamics
- Ropedia W0: Scaling Loco-Manipulation World Models across Embodiments with Human Experience
- Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
- Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC
EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming
First announced by arXiv: 2026-11-13 · Original record ↗
- arxiv
- 2611.16728
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.16728v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Generalist manipulation policies can benefit from the complementary experience of humans, real robots, and simulated agents. However, differences in morphology, sensing, and control interfaces make it difficult to learn effectively from their combined demonstrations. We introduce EgoX, an egocentric cross-embodiment learning framework that organizes heterogeneous demonstrations through semantically corresponding observation and action modalities. EgoX pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on this mixture and fine-tunes it on target-embodiment demonstrations. Within each embodiment, MXT+ assigns a dedicated tokenizer to each observation modality and a dedicated expert to each action modality, with a Transformer trunk shared across embodiments. A training-only embodiment-dreaming objective predicts latent representations of future visual and proprioceptive states, encouraging the shared trunk to model each embodiment's dynamics. Across 13 task-embodiment pairs spanning five tasks and six robot embodiments, including five physical robots and one simulated humanoid, cross-embodiment pretraining raises MXT+'s average normalized task progress from 0.63 to 0.80 over training from scratch. The gains are particularly pronounced under out-of-distribution conditions, where average task progress increases from 0.53 to 0.77. Ablations demonstrate the benefits of broader pretraining mixtures, modular interfaces, and embodiment dreaming, with dreaming producing more similar latent geometry across embodiments. Project page: egox-dream.github.io.
