Driving · 2026-10-15

One-shot Human Video-Guided Adaptation for Vision-Language-Action Models

Qi Chen, Yuwei Wu, Quan Chen, Kunpeng Gao, Jin Chen, Che Sun, Zhi Gao, Yunde Jia

arXiv:2610.19746PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

CapPro is a one-shot adaptation framework for vision-language-action (VLA) models that uses a single human demonstration video to guide a pretrained VLA policy to previously unseen manipulation tasks. It learns phase-aware human-video representations (compact video tokens + phase-caption supervised query tokens) and injects them into the action-conditioning context via a gated adaptation module, without target-task robot demonstrations or fine-tuning.

Why it matters

The authors address limited rapid adaptation of VLA models outside training distributions, where existing methods often need task-specific robot demonstrations and additional fine-tuning. CapPro aims to make cross-task adaptation scalable by using human video guidance instead, while preserving the base VLA policy’s action-generation structure.

Method

  • Phase-aware human video representation: learns a compact demo representation via learned spatial token selection and progress embeddings, plus context-dependent query tokens supervised with phase captions during training.
  • Video-conditioned adaptation: incorporates the compact video representation and query-derived representations into the pretrained VLA policy’s action-conditioning prefix using a learnable gate.
  • One-shot inference: at deployment, uses only current robot observation, language instruction, and a single human demonstration video (no phase captions or target-task robot demonstrations/fine-tuning).

Limitation

Although CapPro improves cross-task generalization, performance on tasks with large object, interaction, and compositional shifts remains limited.

Abstract (from arXiv)

Vision-language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale robot datasets and foundation models for general-purpose robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration video. CapPro consists of two components. First, a phase-aware human video representation module constructs a compact video representation through learned visual token selection and learns task-relevant query representations under phase-caption supervision. Second, a video-conditioned adaptation module jointly incorporates the compact video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-video guidance for cross-task VLA adaptation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC

preprint

One-shot Human Video-Guided Adaptation for Vision-Language-Action Models

Qi Chen, Yuwei Wu, Quan Chen, Kunpeng Gao, Jin Chen, Che Sun, Zhi Gao, Yunde Jia

Preprint posted: 2026-10-14 · Original record ↗

arxiv
2610.19746
Explicit source versions (1)
  • v1 · 2026-10-14
  1. Preprint available

    Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Vision-language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale robot datasets and foundation models for general-purpose robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration video. CapPro consists of two components. First, a phase-aware human video representation module constructs a compact video representation through learned visual token selection and learns task-relevant query representations under phase-caption supervision. Second, a video-conditioned adaptation module jointly incorporates the compact video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-video guidance for cross-task VLA adaptation.