Trace and evaluate agent workflows with OpenTelemetry/OpenInference instrumentation and LLM-as-a-judge
Instrument an agent so each step (inputs, tool calls, and outputs) is traced to a centralized UI, then run targeted evaluations over the captured tool executions using an LLM-as-a-judge to score properties like retrieval relevance.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Tracing makes it possible to inspect an agent’s step-by-step behavior and debug where it goes wrong. Evaluation then provides a systematic way to check whether intermediate decisions (e.g., tool outputs) are actually effective, not merely successful.
What changed
Instead of treating the agent as a black box, the workflow adds (1) tracing via OpenTelemetry plus OpenInference instrumentation to visualize calls end-to-end and (2) eval logging based on LLM judge classifications of traced tool outputs.
Where it helps
- Debugging multi-step tool-calling agents by inspecting tool invocation traces
- Evaluating retrieval quality by judging tool search outputs for relevance
- Logging and comparing agent run quality over repeated executions
- Assessing whether intermediate retrieved information aligns with the query intent
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The described relevance evaluation depends on an LLM judge and the specific prompt/template used to classify outputs; classification quality is therefore sensitive to the judge setup and prompt choice.
Sources
- Trace & Evaluate your Agent with Arize Phoenix ↗
Hugging Face · primary · Published 2025-02-28 · Checked 2026-11-10
