Driving · 2026-11-04
Controlling Action-Head Co-Adaptation to VLM Representation Anisotropy
Ruijie Zhang, Jiayin Zheng, Yequan Zhao, Liyan Tan, Ziyue Liu, Zhengyang Wang, Kunpeng Song, Nathan Zhao, Jian Yao, Yin Zhang, Zheng Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study how an action head co-adapts to anisotropic channel variance in pretrained VLM representations in vision-language-action (VLA) models. They observe that head sensitivity progressively concentrates on high-variance VLM dimensions. They propose AdaIE, a parameter-free interface reweighting method based on online sensitivity estimates, improving success rates on RoboCasa-GR1, RoboTwin 2.0, real-world manipulation, and also generalizing to diffusion policies and world models.
Why it matters
When pretrained representations are connected to downstream action heads, the interface geometry can shape what the head learns. The authors show that under representation–task mismatch, the head’s sensitivity concentration can hurt performance, and that counteracting it with AdaIE improves downstream results. AdaIE directly targets this co-adaptation problem at the backbone–head interface via sensitivity-based reweighting.
Method
- Analyze VLM–DiT/VLA interfaces by tracking channel-wise VLM representation variance and action-head sensitivity (estimated via Jacobian diagonal probing).
- Controlled study: sensitivity reweighting via exponent α (Equalize vs Exploit) shows consistent gains under representation–task mismatch.
- Propose AdaIE: parameter-free, online sensitivity estimation with Hutchinson probes and inverse sensitivity reweighting plus norm-matching to regulate evolving interface co-adaptation.
Abstract (from arXiv)
Modern learning systems increasingly combine pretrained models with downstream heads. Despite the growing importance of such modular designs, how the downstream model co-adapts to the geometry of pretrained representations remains relatively underexplored. We study this problem through vision-language-action models (VLAs), where a pretrained vision-language model (VLM) conditions an action model for control. We find that the action model progressively concentrates its sensitivity on high-variance dimensions of the VLM outputs. Through controlled experiments across different backbone-head models, we further show that counteracting this sensitivity concentration can be beneficial under representation--task mismatch settings. Motivated by these observations, we introduce \emph{Adaptive Interface Equalizer} (AdaIE), a parameter-free mechanism that reweights pretrained representations according to the evolving downstream sensitivity during training. AdaIE improves VLA success rates on simulation benchmarks, with gains of up to $+5.28$ percentage points (pp) on RoboCasa-GR1 and $+1.68$ pp on RoboTwin 2.0, and achieves an average gain of $+19$ pp across five real-world manipulation tasks. It further generalizes to diffusion policies and world models, improving downstream performance in both settings. Our results highlight interface co-adaptation as an important optimization problem when coupling pretrained representations with downstream models.
Related papers
- Causal Analysis of Robot Foundation Models Enables Efficiency Gains
- One-shot Human Video-Guided Adaptation for Vision-Language-Action Models
- Axiom-0: Hierarchical Self-Refinement for Vision-Language-Action Policies
- SWAM: Scaling World Action Models with Cross-Modal Alignment
- Juno: Taming Predictive Latents for Vision-Language-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC
Controlling Action-Head Co-Adaptation to VLM Representation Anisotropy
First announced by arXiv: 2026-11-04 · Original record ↗
- arxiv
- 2611.04006
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.04006v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Modern learning systems increasingly combine pretrained models with downstream heads. Despite the growing importance of such modular designs, how the downstream model co-adapts to the geometry of pretrained representations remains relatively underexplored. We study this problem through vision-language-action models (VLAs), where a pretrained vision-language model (VLM) conditions an action model for control. We find that the action model progressively concentrates its sensitivity on high-variance dimensions of the VLM outputs. Through controlled experiments across different backbone-head models, we further show that counteracting this sensitivity concentration can be beneficial under representation--task mismatch settings. Motivated by these observations, we introduce \emph{Adaptive Interface Equalizer} (AdaIE), a parameter-free mechanism that reweights pretrained representations according to the evolving downstream sensitivity during training. AdaIE improves VLA success rates on simulation benchmarks, with gains of up to $+5.28$ percentage points (pp) on RoboCasa-GR1 and $+1.68$ pp on RoboTwin 2.0, and achieves an average gain of $+19$ pp across five real-world manipulation tasks. It further generalizes to diffusion policies and world models, improving downstream performance in both settings. Our results highlight interface co-adaptation as an important optimization problem when coupling pretrained representations with downstream models.
