← AI Techniques Radar
Reasoning & planning

Trace and evaluate agent workflows with OpenTelemetry/OpenInference instrumentation and LLM-as-a-judge

Instrument an agent so each step (inputs, tool calls, and outputs) is traced to a centralized UI, then run targeted evaluations over the captured tool executions using an LLM-as-a-judge to score properties like retrieval relevance.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

Tracing makes it possible to inspect an agent’s step-by-step behavior and debug where it goes wrong. Evaluation then provides a systematic way to check whether intermediate decisions (e.g., tool outputs) are actually effective, not merely successful.

What changed

Instead of treating the agent as a black box, the workflow adds (1) tracing via OpenTelemetry plus OpenInference instrumentation to visualize calls end-to-end and (2) eval logging based on LLM judge classifications of traced tool outputs.

Where it helps

  • Debugging multi-step tool-calling agents by inspecting tool invocation traces
  • Evaluating retrieval quality by judging tool search outputs for relevance
  • Logging and comparing agent run quality over repeated executions
  • Assessing whether intermediate retrieved information aligns with the query intent

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The described relevance evaluation depends on an LLM judge and the specific prompt/template used to classify outputs; classification quality is therefore sensitive to the judge setup and prompt choice.

Sources