← AI Techniques Radar
Inference efficiency

EVA-style end-to-end bot-to-bot audio evaluation with dual-dimension scoring

Evaluate conversational voice agents by running complete multi-turn spoken interactions in a bot-to-bot audio simulation, then score each dialogue along two coupled dimensions: task accuracy and spoken conversational experience. Use deterministic, tool-log-based checks for objective completion, plus LLM/LALM judge metrics for faithfulness and audio-level speech fidelity, and experience metrics for conciseness, progression, and turn-taking timing.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

Voice agent quality depends on more than reaching the right final state; the work also claims that failures like mishearing key tokens, fabricating policy details, or producing overly verbose/interrupting dialogue can appear invisible to task-only benchmarks. Joint scoring is intended to expose these accuracy–experience tradeoffs.

What changed

Moves from component- or post-hoc evaluation (e.g., separate speech recognition quality, or conversation behavior assessed without tool-completed workflows) to end-to-end evaluation of realistic multi-turn audio conversations that include tool orchestration and a verifiable final state. Introduces dual high-level scoring (Accuracy vs Experience) and diagnostic metrics evaluated within whole dialogues rather than isolated modules.

Where it helps

  • Benchmarking conversational voice agents that must both complete tasks and speak naturally
  • Diagnosing failures from multi-step tool workflows in spoken dialogues
  • Comparing cascade vs audio-native voice agent architectures under the same audio task simulation
  • Evaluating spoken entity-critical outputs like confirmation codes and prices

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

Reported limitations include potential bias from LLM-as-judge models and the possibility that task completion is treated as binary rather than graded. The current setup is based on a synthetic, single-domain (airline) scenario set and a simulated caller that may not perfectly match real callers; reproducing some elements can depend on commercial API access, and simulation fidelity may vary by provider/infrastructure.

Sources