Robotics · 2026-11-02
Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space
Biaoliang Guan, Tingyu Yuan, Wen Ye, Ziming Ji, Xu Yang, Yan Huang, Jihua Zhu, Chaoyang Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Nexus-VLA is a hybrid human-robot policy learning framework that transfers manipulation knowledge across embodiments via a hierarchical semantic latent. It end-to-end co-trains on unpaired human and robot demonstrations, progressively aligning human/robot semantics at four levels (Intent, Spatial, Motion, Interaction) and shaping a shared latent used directly for continuous action generation. The authors report 80.5% task success on RoboCasa-GR1 and strong OOD and real-world generalization.
Why it matters
The authors address the embodiment gap between human and robot video experiences (visual appearance, kinematics, action spaces) that limits effective knowledge transfer. By unifying hierarchical semantic alignment and joint end-to-end training, Nexus-VLA aims to let human manipulation experience directly contribute to robot action learning, improving human-data utilization, learning efficiency, and cross-scenario generalization.
Method
- Introduces hierarchical cross-embodiment semantic alignment across four levels: Intent, Spatial, Motion, Interaction, progressively grounded coarse-to-fine.
- Builds a learnable Hierarchical Semantic Latent (final hidden state of a semantic token) and shapes it using multi-level semantic prediction objectives; intermediate semantics are not needed at inference.
- Performs end-to-end human-robot mixed training with dual-space action representation (shared wrist/end-effector motion + robot joint control) and conditional flow-matching action generation.
Abstract (from arXiv)
Large-scale egocentric human videos provide rich and scalable manipulation experience for robot policy learning, yet substantial embodiment gaps in visual appearance, kinematic structure, and action spaces hinder effective knowledge transfer. Existing methods often decouple cross-embodiment knowledge alignment from robot policy learning, making it difficult for human manipulation knowledge to directly and fully contribute to robot action learning. To address this issue, we propose Nexus-VLA, a hybrid policy learning framework that enables cross-embodiment knowledge transfer through end-to-end human-robot joint training, allowing human manipulation experience to directly contribute to robot policy learning. We first introduce a hierarchical cross-embodiment semantic alignment paradigm that progressively establishes shared manipulation semantics from task intent to fine-grained interactions, enabling more complete use of the manipulation knowledge contained in human videos. Building on this hierarchy, we design a shared manipulation latent representation that jointly encodes multi-level shared knowledge into an internal representation directly used for continuous action generation, improving policy learning efficiency while effectively leveraging human knowledge. Across RoboCasa-GR1, OOD settings, and real-world robot experiments, Nexus-VLA achieves an 80.5% task success rate on RoboCasa-GR1 and demonstrates strong advantages in human-data utilization, learning efficiency, and cross-scenario generalization.
Related papers
- World Action Learning via Interaction-Centric Spectral Latent Guidance
- Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation
- SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-30 · Indexed by Pub-AI: 2026-11-02 UTC
Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space
Preprint posted: 2026-10-30 · Original record ↗
- arxiv
- 2610.42021
Explicit source versions (1)
- v1 · 2026-10-30
- Preprint available
Source date: 2026-10-30 · First detected: 2026-11-02 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.42021v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Large-scale egocentric human videos provide rich and scalable manipulation experience for robot policy learning, yet substantial embodiment gaps in visual appearance, kinematic structure, and action spaces hinder effective knowledge transfer. Existing methods often decouple cross-embodiment knowledge alignment from robot policy learning, making it difficult for human manipulation knowledge to directly and fully contribute to robot action learning. To address this issue, we propose Nexus-VLA, a hybrid policy learning framework that enables cross-embodiment knowledge transfer through end-to-end human-robot joint training, allowing human manipulation experience to directly contribute to robot policy learning. We first introduce a hierarchical cross-embodiment semantic alignment paradigm that progressively establishes shared manipulation semantics from task intent to fine-grained interactions, enabling more complete use of the manipulation knowledge contained in human videos. Building on this hierarchy, we design a shared manipulation latent representation that jointly encodes multi-level shared knowledge into an internal representation directly used for continuous action generation, improving policy learning efficiency while effectively leveraging human knowledge. Across RoboCasa-GR1, OOD settings, and real-world robot experiments, Nexus-VLA achieves an 80.5% task success rate on RoboCasa-GR1 and demonstrates strong advantages in human-data utilization, learning efficiency, and cross-scenario generalization.
