Unassigned · 2026-10-05

CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges

Hoda Ayad, Tanu Mitra, Abhishek Mukherji

arXiv:2610.02622PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose CuBEs (Culturally-situated Behavioral Evaluations), extending an automated evaluation pipeline to inject cultural context into scenario generation and automated judging. Using a human-labeled dataset across 12 cultures and evaluating 13 LLMs, they report significant cross-cultural variation in the presence of six risky behaviors and show that culturally-agnostic evaluations can miss or invert these effects.

Why it matters

The paper argues that behavioral evaluations that ignore cultural context lack generalizability for global deployments. The authors report that adding cultural situatedness changes measured behavior presence across six behaviors and that a no-culture baseline can substantially overstate or understate model behavior, even flipping verdicts (e.g., political bias) between “strongly present” and “absent.”

Method

  • Extend Anthropic’s Bloom (agentic) behavior-evaluation pipeline to inject localized cultural context across behavior formulation, scenario generation, and evaluation.
  • Optimize pipeline components via experiments: scenario generation model selection (pairwise human preference), user simulation model selection (affordability vs elicitation), and judge alignment (agreement with human majority across cultures).
  • Run cross-cultural evaluations across behaviors, cultures, domains, and 13 open/closed-source LLMs; judge behavior presence on a 1–10 scale.

Limitation

Automated judges are still a ways away from capturing true cultural nuance: the paper states they tend to default to a culturally agnostic “modal” perspective, flattening cross-cultural label variance and under-shifting responses compared to localized human annotators.

Abstract (from arXiv)

Evaluating the occurrence and triggers of large language model (LLM) behaviors - such as sycophancy, self-preference, or over-confidence - is critical for predicting real-world model deployment risks. However, existing situated behavioral evaluations typically ignore cultural context, limiting their generalizability across an increasingly global user base. To address this gap, we propose CuBEs - Culturally-situated Behavior Evaluations that probe for response patterns across diverse user cultures. We first extend an automated testing pipeline to inject cultural context into behavioral test scenarios and subsequent evaluation. We assess the cultural adaptability of this pipeline by building a human-labeled dataset that captures nuanced dimensions of behavior understanding across 12 distinct cultures. Our dataset reveals significant cross-cultural variations that one-size-fits all judgments fail to capture. Through evaluating 13 open- and closed-source LLMs, we find that introducing cultural situatedness in the evaluation scenario creates significant variation in the presence of a behavior. For example, while our baseline experiments testing for political bias capture localized Western political dimensions like the American conservative-progressive divide, non-Western culturally situated evaluations surface entirely different axes of bias such as religious and colonial political issues. Our findings demonstrate that standard, culturally-agnostic evaluations fail to capture these shifts, highlighting the necessity of culturally situated behavioral testing for global deployments.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC

preprint

CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges

Hoda Ayad, Tanu Mitra, Abhishek Mukherji

Preprint posted: 2026-10-02 · Original record ↗

arxiv
2610.02622
Explicit source versions (1)
  • v1 · 2026-10-02
  1. Preprint available

    Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Evaluating the occurrence and triggers of large language model (LLM) behaviors - such as sycophancy, self-preference, or over-confidence - is critical for predicting real-world model deployment risks. However, existing situated behavioral evaluations typically ignore cultural context, limiting their generalizability across an increasingly global user base. To address this gap, we propose CuBEs - Culturally-situated Behavior Evaluations that probe for response patterns across diverse user cultures. We first extend an automated testing pipeline to inject cultural context into behavioral test scenarios and subsequent evaluation. We assess the cultural adaptability of this pipeline by building a human-labeled dataset that captures nuanced dimensions of behavior understanding across 12 distinct cultures. Our dataset reveals significant cross-cultural variations that one-size-fits all judgments fail to capture. Through evaluating 13 open- and closed-source LLMs, we find that introducing cultural situatedness in the evaluation scenario creates significant variation in the presence of a behavior. For example, while our baseline experiments testing for political bias capture localized Western political dimensions like the American conservative-progressive divide, non-Western culturally situated evaluations surface entirely different axes of bias such as religious and colonial political issues. Our findings demonstrate that standard, culturally-agnostic evaluations fail to capture these shifts, highlighting the necessity of culturally situated behavioral testing for global deployments.