Robotics · 2026-11-02

Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space

Biaoliang Guan, Tingyu Yuan, Wen Ye, Ziming Ji, Xu Yang, Yan Huang, Jihua Zhu, Chaoyang Zhao

arXiv:2610.42021PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Nexus-VLA is a hybrid human-robot policy learning framework that transfers manipulation knowledge across embodiments via a hierarchical semantic latent. It end-to-end co-trains on unpaired human and robot demonstrations, progressively aligning human/robot semantics at four levels (Intent, Spatial, Motion, Interaction) and shaping a shared latent used directly for continuous action generation. The authors report 80.5% task success on RoboCasa-GR1 and strong OOD and real-world generalization.

Why it matters

The authors address the embodiment gap between human and robot video experiences (visual appearance, kinematics, action spaces) that limits effective knowledge transfer. By unifying hierarchical semantic alignment and joint end-to-end training, Nexus-VLA aims to let human manipulation experience directly contribute to robot action learning, improving human-data utilization, learning efficiency, and cross-scenario generalization.

Method

  • Introduces hierarchical cross-embodiment semantic alignment across four levels: Intent, Spatial, Motion, Interaction, progressively grounded coarse-to-fine.
  • Builds a learnable Hierarchical Semantic Latent (final hidden state of a semantic token) and shapes it using multi-level semantic prediction objectives; intermediate semantics are not needed at inference.
  • Performs end-to-end human-robot mixed training with dual-space action representation (shared wrist/end-effector motion + robot joint control) and conditional flow-matching action generation.
Abstract (from arXiv)

Large-scale egocentric human videos provide rich and scalable manipulation experience for robot policy learning, yet substantial embodiment gaps in visual appearance, kinematic structure, and action spaces hinder effective knowledge transfer. Existing methods often decouple cross-embodiment knowledge alignment from robot policy learning, making it difficult for human manipulation knowledge to directly and fully contribute to robot action learning. To address this issue, we propose Nexus-VLA, a hybrid policy learning framework that enables cross-embodiment knowledge transfer through end-to-end human-robot joint training, allowing human manipulation experience to directly contribute to robot policy learning. We first introduce a hierarchical cross-embodiment semantic alignment paradigm that progressively establishes shared manipulation semantics from task intent to fine-grained interactions, enabling more complete use of the manipulation knowledge contained in human videos. Building on this hierarchy, we design a shared manipulation latent representation that jointly encodes multi-level shared knowledge into an internal representation directly used for continuous action generation, improving policy learning efficiency while effectively leveraging human knowledge. Across RoboCasa-GR1, OOD settings, and real-world robot experiments, Nexus-VLA achieves an 80.5% task success rate on RoboCasa-GR1 and demonstrates strong advantages in human-data utilization, learning efficiency, and cross-scenario generalization.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-30 · Indexed by Pub-AI: 2026-11-02 UTC

preprint

Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space

Biaoliang Guan, Tingyu Yuan, Wen Ye, Ziming Ji, Xu Yang, Yan Huang, Jihua Zhu, Chaoyang Zhao

Preprint posted: 2026-10-30 · Original record ↗

arxiv
2610.42021
Explicit source versions (1)
  • v1 · 2026-10-30
  1. Preprint available

    Source date: 2026-10-30 · First detected: 2026-11-02 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Large-scale egocentric human videos provide rich and scalable manipulation experience for robot policy learning, yet substantial embodiment gaps in visual appearance, kinematic structure, and action spaces hinder effective knowledge transfer. Existing methods often decouple cross-embodiment knowledge alignment from robot policy learning, making it difficult for human manipulation knowledge to directly and fully contribute to robot action learning. To address this issue, we propose Nexus-VLA, a hybrid policy learning framework that enables cross-embodiment knowledge transfer through end-to-end human-robot joint training, allowing human manipulation experience to directly contribute to robot policy learning. We first introduce a hierarchical cross-embodiment semantic alignment paradigm that progressively establishes shared manipulation semantics from task intent to fine-grained interactions, enabling more complete use of the manipulation knowledge contained in human videos. Building on this hierarchy, we design a shared manipulation latent representation that jointly encodes multi-level shared knowledge into an internal representation directly used for continuous action generation, improving policy learning efficiency while effectively leveraging human knowledge. Across RoboCasa-GR1, OOD settings, and real-world robot experiments, Nexus-VLA achieves an 80.5% task success rate on RoboCasa-GR1 and demonstrates strong advantages in human-data utilization, learning efficiency, and cross-scenario generalization.