Robotics · 2026-10-22

Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI

Han Hu, He Chen

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors evaluate seven locally deployable VLMs on two person-centric thermal HRI datasets. They test fine-grained action recognition, action–object interaction and temporal grounding, and candidate-set backoff/abstention. Models recognize broad activity categories but struggle on fine-grained actions, produce inaccurate event timelines, and hallucinate object-related actions. Intervention gains from input preprocessing/reasoning are limited and model abstention remains unreliable.

Why it matters

For thermal human activity understanding in robotics, it is not enough to know that models “work” on averages. The authors show recurring fine-action confusions, timeline errors, and hallucinated object interactions, plus inconsistent abstention behavior even when explicitly instructed. These failure modes limit reliability for practical robotic use in low-light and privacy-sensitive settings.

Method

Evaluate seven locally deployable VLMs (no task-specific adaptation) on two person-centric thermal datasets with three diagnostic tasks: fine-grained action recognition, interaction recognition with temporal grounding, and candidate-set backoff.
Use paired RGB–thermal recordings as a performance reference, and run controlled variations in frame sampling, thermal input transformations, and reasoning; also test temporal-evidence interventions.
Measure abstention/backoff using paired GT-present vs GT-absent questions with an explicit instruction to back off when evidence is insufficient.

Limitation

The authors state that their tasks are behavioral proxies for robotic applications and that a thermal-video benchmark evaluating whether a robot responds appropriately to human activity is still missing.

Abstract (from arXiv)

Thermal cameras enable robots to perceive human activities in darkness while revealing less personally identifiable appearance information and fewer details of the surrounding environment than RGB cameras. As vision-language models (VLMs), originally developed for RGB imagery, are increasingly integrated into embodied systems for scene understanding and high-level planning, thermal sensing offers a promising way to extend activity perception to low-light and privacy-sensitive settings. However, the failure modes of VLMs in thermal human activity understanding for human-robot interaction (HRI) remain underexplored. We fill this gap by evaluating seven locally deployable VLMs, including models developed for physical AI and robotics, on two person-centric thermal datasets. Our diagnostic tasks examine fine-grained action recognition, activity timeline reconstruction, and the ability to withhold unsupported answers. Paired RGB recordings serve as a performance reference, while controlled variations in frame sampling, thermal input presentation, and reasoning are used to assess whether the observed errors can be mitigated. Our results show that models can recognize broad activity categories but struggle with fine-grained actions such as coughing, sneezing, and yawning. They also produce inaccurate event timelines and hallucinate actions involving objects. Simple thermal input preprocessing and additional reasoning offer only limited, model-dependent gains. Abstention behavior also varies across models even under explicit instructions, limiting their reliability for practical robotic use.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC

preprint