Unassigned · 2026-10-20

The Explainability Illusion: Must ViTs Exhibit Patch-Decoding Interpretability?

Ronen Schaffer, Hila Manor, Tomer Michaeli

arXiv:2610.25289PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study patch-token (token-decoding) explainability in Vision Transformers and find it is not guaranteed. Across many real-world ViTs, some models show aligned, others inverted, and some unexplainable maps, and the behavior can change with the optimization landscape (including task/data/optimization choices). They also analyze contrastive ViTs/VLMs and provide theory showing a toy transformer can implement the same optimal rule with different patch-decoding interpretability patterns.

Why it matters

Patch-token decoding is widely used as a diagnostic for transformer interpretability. The authors report that this “token-decoding explainability” can be inverted or absent and is not an inherent property of trained ViTs. Their results suggest that patch-token decoding should not be assumed to reliably localize semantics without checking the model- and setting-dependent regime.

Method

  • Defined aligned-, inverted-, and unexplainable explainability regimes using the relationship between mean patch scores in an object mask vs its complement (PartImageNet++ evaluations).
  • Analyzed many pre-trained ViTs and contrastive ViTs/VLMs with zero-shot patch-token decoding across architectures, scales, objectives, and training losses; included controlled synthetic experiments (described).
  • Provided theoretical support by proving a setting where a single toy transformer can implement an optimal classification rule in multiple weight configurations with different patch-decoding explainability patterns.

Limitation

The provided text does not state a specific limitation of the study.

Abstract (from arXiv)

Transformers are widely assumed to exhibit token-decoding explainability. In particular, in vision transformer (ViT) classifiers, decoding individual patch tokens has become a common diagnostic tool due to its tendency to yield human-interpretable outputs. In light of the increasing popularity of this approach, an important question is whether token-decoding explainability is an inherent property of transformers that should be expected to hold across architectures, scales, and training objectives, or merely an artifact of some implicit bias. In this paper we systematically study token-decoding explainability across diverse ViT settings, spanning architectures, scales, and training objectives, using both controlled experiments and real-world models. We find that while some models indeed exhibit aligned and interpretable token decodings, others exhibit either inverted explainability maps or a complete lack of correlation with the image semantics. As we show, these tendencies toward different explainability behaviors are not intrinsic to the architecture and can vary arbitrarily with changes in the optimization landscape. We provide theoretical support for these observations by demonstrating a setting where a single fixed toy transformer architecture can provably implement the optimal classification rule in multiple ways, which differ in their patch-decoding explainability patterns. Our results show that token-decoding explainability is not an inherent property of trained transformers, but rather an outcome contingent on the problem setting and thus should not be a-priori assumed to hold. Code can be found in https://ronensc.github.io/Explainability-Illusion/ .

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-18 · Indexed by Pub-AI: 2026-10-20 UTC

preprint

The Explainability Illusion: Must ViTs Exhibit Patch-Decoding Interpretability?

Ronen Schaffer, Hila Manor, Tomer Michaeli

Preprint posted: 2026-10-18 · Original record ↗

arxiv
2610.25289
Explicit source versions (1)
  • v1 · 2026-10-18
  1. Preprint available

    Source date: 2026-10-18 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Transformers are widely assumed to exhibit token-decoding explainability. In particular, in vision transformer (ViT) classifiers, decoding individual patch tokens has become a common diagnostic tool due to its tendency to yield human-interpretable outputs. In light of the increasing popularity of this approach, an important question is whether token-decoding explainability is an inherent property of transformers that should be expected to hold across architectures, scales, and training objectives, or merely an artifact of some implicit bias. In this paper we systematically study token-decoding explainability across diverse ViT settings, spanning architectures, scales, and training objectives, using both controlled experiments and real-world models. We find that while some models indeed exhibit aligned and interpretable token decodings, others exhibit either inverted explainability maps or a complete lack of correlation with the image semantics. As we show, these tendencies toward different explainability behaviors are not intrinsic to the architecture and can vary arbitrarily with changes in the optimization landscape. We provide theoretical support for these observations by demonstrating a setting where a single fixed toy transformer architecture can provably implement the optimal classification rule in multiple ways, which differ in their patch-decoding explainability patterns. Our results show that token-decoding explainability is not an inherent property of trained transformers, but rather an outcome contingent on the problem setting and thus should not be a-priori assumed to hold. Code can be found in https://ronensc.github.io/Explainability-Illusion/ .