HALMES: selectively triggering contrastive decoding using hallucination-precursor attention signals
HALMES is a lightweight, plug-and-play mechanism for large vision-language models that monitors internal attention features during generation to detect hallucination-precursor token segments. Instead of applying contrastive decoding uniformly at every decoding step, HALMES selectively activates contrastive decoding only when hallucination-prone segments are identified, aiming to correct likely hallucinations while preserving otherwise correct outputs.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Author-claimed motivation is that full-stage (every-step) contrastive decoding can harm correct generations and may even introduce new hallucinations. By intervening only when precursor abnormalities appear, HALMES aims to reduce negative side effects while improving hallucination mitigation quality.
What changed
The change is from indiscriminate, continuous contrastive-decoding intervention at all decoding steps to an on-demand approach: detect hallucination precursor segments from attention features and trigger contrastive decoding selectively for those spans.
Where it helps
- Hallucination mitigation in vision-language text generation
- On-demand integration with contrastive-decoding inference-time strategies
- Reducing disruption to correct generations when using contrastive decoding
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Pub-AI has not independently evaluated this method. Consult the linked sources for their evaluation setup and limitations.
Sources
- HALMES: Knowing When to Intervene in LVLM Hallucination Mitigation ↗
arXiv · primary · Checked 2026-11-10
