Your agent eval says accuracy improved. But did latency spike? Does your LLM-based metric even agree with human judgment? And is that 5% gain real or noise? Do we ship it or not?
If you’re evaluating AI agents, you’ve likely encountered hidden failures such as:

1. Improving accuracy with a tool change also increases tool calls and latency, and a single positive metric masks overall degradation.
2. LLM-based evaluators are nondeterministic, so a score increase may only reflect sensitivity to a prompt change, not an improved user experience.
3. Without robust testing, you might be shipping coin-flip gains that will disappear on the next run.

In this session, I’ll walk through how we addressed these at Elastic. Using a real experiment as an example, I’ll cover the evaluation setup we built to catch these failures. This includes multi-metric evaluation to expose tradeoffs (accuracy, tool usage, and latency) and a claim-level correctness evaluator (we developed in house) validated against human judgment to ensure LLM-based scores are meaningful. I’ll also discuss key significance testing principles we used to filter out noise and verify real gains.

Along the way, I’ll show the prompt structure behind our evaluator and examples of practical results.