Driving · 2026-10-15
One-shot Human Video-Guided Adaptation for Vision-Language-Action Models
Qi Chen, Yuwei Wu, Quan Chen, Kunpeng Gao, Jin Chen, Che Sun, Zhi Gao, Yunde Jia
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CapPro is a one-shot adaptation framework for vision-language-action (VLA) models that uses a single human demonstration video to guide a pretrained VLA policy to previously unseen manipulation tasks. It learns phase-aware human-video representations (compact video tokens + phase-caption supervised query tokens) and injects them into the action-conditioning context via a gated adaptation module, without target-task robot demonstrations or fine-tuning.
Why it matters
The authors address limited rapid adaptation of VLA models outside training distributions, where existing methods often need task-specific robot demonstrations and additional fine-tuning. CapPro aims to make cross-task adaptation scalable by using human video guidance instead, while preserving the base VLA policy’s action-generation structure.
Method
- Phase-aware human video representation: learns a compact demo representation via learned spatial token selection and progress embeddings, plus context-dependent query tokens supervised with phase captions during training.
- Video-conditioned adaptation: incorporates the compact video representation and query-derived representations into the pretrained VLA policy’s action-conditioning prefix using a learnable gate.
- One-shot inference: at deployment, uses only current robot observation, language instruction, and a single human demonstration video (no phase captions or target-task robot demonstrations/fine-tuning).
Limitation
Although CapPro improves cross-task generalization, performance on tasks with large object, interaction, and compositional shifts remains limited.
Abstract (from arXiv)
Vision-language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale robot datasets and foundation models for general-purpose robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration video. CapPro consists of two components. First, a phase-aware human video representation module constructs a compact video representation through learned visual token selection and learns task-relevant query representations under phase-caption supervision. Second, a video-conditioned adaptation module jointly incorporates the compact video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-video guidance for cross-task VLA adaptation.
Related papers
- PaceVLA: Adaptive Temporal Action Quantization in Vision-Language-Action Models
- APS-VLA: Action-Guided and Future-Aware Token Pruning for Efficient Vision-Language-Action Models
- Controlling Action-Head Co-Adaptation to VLM Representation Anisotropy
- Axiom-0: Hierarchical Self-Refinement for Vision-Language-Action Policies
- PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
One-shot Human Video-Guided Adaptation for Vision-Language-Action Models
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.19746
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale robot datasets and foundation models for general-purpose robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration video. CapPro consists of two components. First, a phase-aware human video representation module constructs a compact video representation through learned visual token selection and learns task-relevant query representations under phase-caption supervision. Second, a video-conditioned adaptation module jointly incorporates the compact video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-video guidance for cross-task VLA adaptation.
