EVA-style end-to-end bot-to-bot audio evaluation with dual-dimension scoring
Evaluate conversational voice agents by running complete multi-turn spoken interactions in a bot-to-bot audio simulation, then score each dialogue along two coupled dimensions: task accuracy and spoken conversational experience. Use deterministic, tool-log-based checks for objective completion, plus LLM/LALM judge metrics for faithfulness and audio-level speech fidelity, and experience metrics for conciseness, progression, and turn-taking timing.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Voice agent quality depends on more than reaching the right final state; the work also claims that failures like mishearing key tokens, fabricating policy details, or producing overly verbose/interrupting dialogue can appear invisible to task-only benchmarks. Joint scoring is intended to expose these accuracy–experience tradeoffs.
What changed
Moves from component- or post-hoc evaluation (e.g., separate speech recognition quality, or conversation behavior assessed without tool-completed workflows) to end-to-end evaluation of realistic multi-turn audio conversations that include tool orchestration and a verifiable final state. Introduces dual high-level scoring (Accuracy vs Experience) and diagnostic metrics evaluated within whole dialogues rather than isolated modules.
Where it helps
- Benchmarking conversational voice agents that must both complete tasks and speak naturally
- Diagnosing failures from multi-step tool workflows in spoken dialogues
- Comparing cascade vs audio-native voice agent architectures under the same audio task simulation
- Evaluating spoken entity-critical outputs like confirmation codes and prices
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Reported limitations include potential bias from LLM-as-judge models and the possibility that task completion is treated as binary rather than graded. The current setup is based on a synthetic, single-domain (airline) scenario set and a simulated caller that may not perfectly match real callers; reproducing some elements can depend on commercial API access, and simulation fidelity may vary by provider/infrastructure.
Sources
- A New Framework for Evaluating Voice Agents (EVA) ↗
Hugging Face · primary · Published 2026-03-24 · Checked 2026-11-10
