Driving · 2026-10-05
ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models
Hainiu Xu, V\'{i}tor N. Louren\c{c}o, Mohnish Dubey, Yunfei Bai, Yulan He, Caroline Catmur, Aline Paes, Marco Caserta, Akash Chandrayan, Luca D'Angelo
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ReFract is a benchmark (150 expert-validated entries) for “perspective awareness” in LLM language model agents: agents must take role-calibrated actions and tool-calling trajectories that respect both capability and knowledge boundaries. Using text world models that gate tools by role, the authors report that state-of-the-art LLMs solve at most 69% of tasks and more than 50% of trajectories contain perspective-violating actions.
Why it matters
The paper argues that high-stakes environments (e.g., industrial maintenance) make persona-mismatched actions potentially irreversible, so evaluation should measure not just task completion but whether an agent infers a role’s intentions and acts through tools appropriate to that role. The authors introduce Perspective-Taking and Perspective-Routing and ground evaluation in executable per-persona text world models, enabling process- and trajectory-level assessment of perspective awareness.
Method
- Introduce ReFract: 150 expert-validated entries where the same query requires different role-dependent action trajectories.
- Construct executable Text World Models (Python programs) with tool functions gated by per-role capability and knowledge; trajectories are evaluated against persona constraints at every step.
- Evaluate LLMs under Query-Specific Tool Set (QTS) vs Full Tool Set (FTS) regimes and Function Calling vs ReAct scaffolds; report pass rate, persona compliance, over/under reach, and efficiency.
Limitation
The authors exclude Claude models from the baseline evaluation because Claude models are heavily involved in the generation and validation of ReFract, to mitigate circularity.
Abstract (from arXiv)
Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles. A capable agent must therefore act in a way that is calibrated to user's role: taking actions and providing information that respect the role's knowledge and capability boundaries. Unlike coding, where mistakes are usually recoverable, agent responses in these settings are enacted on physical equipment, and can therefore cause irreversible equipment damage, production loss, or personnel harm. Existing benchmarks, however, largely overlook the need for agents to infer what a role intends and acting only through tools that role may legitimately use, a capability which we term Perspective Awareness. To this end, we introduce ReFract, a benchmark of 150 expert-validated entries in which an agent must act differently in response to the same query depending on user's role. Entries of ReFract are grounded in anonymized queries from domain support conversations, against which we construct Text World Models that simulate the agent's operating environments and assemble perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of the tasks with more than 50% of their trajectories contain attempts of taking perspective-violating actions. ReFract exposes perspective awareness as a distinct, largely unsolved axis of agent evaluation and motivates agents that calibrate not just how to act, but for whom.
Related papers
- Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
- LAW: Unifying Language, Action, and World Modeling for Agentic Physical Intelligence
- Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
- AgentVQA: A Unified Benchmark for Agentic Visual Understanding
- SEAR: Self-Evolving Agentic Robotics
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.03356
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles. A capable agent must therefore act in a way that is calibrated to user's role: taking actions and providing information that respect the role's knowledge and capability boundaries. Unlike coding, where mistakes are usually recoverable, agent responses in these settings are enacted on physical equipment, and can therefore cause irreversible equipment damage, production loss, or personnel harm. Existing benchmarks, however, largely overlook the need for agents to infer what a role intends and acting only through tools that role may legitimately use, a capability which we term Perspective Awareness. To this end, we introduce ReFract, a benchmark of 150 expert-validated entries in which an agent must act differently in response to the same query depending on user's role. Entries of ReFract are grounded in anonymized queries from domain support conversations, against which we construct Text World Models that simulate the agent's operating environments and assemble perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of the tasks with more than 50% of their trajectories contain attempts of taking perspective-violating actions. ReFract exposes perspective awareness as a distinct, largely unsolved axis of agent evaluation and motivates agents that calibrate not just how to act, but for whom.