How to Evaluate AI Agents

Evaluating an agent is a problem of distinguishing between different outcomes of the same apparent interaction. When an agent uses tools, changes records, and acts on behalf of a user, its final message is only one piece of evidence.

You face a recurring decision: how to design an evaluation procedure that provides confidence in a system that acts over time. A useful evaluation establishes whether the agent respected instructions, handled uncertainty, and used resources within acceptable limits.

Start with the change the user wanted#

Broad goals like “handle refunds well” are impossible to measure. You need a concrete requirement, such as “issue exactly one refund for the correct amount on an eligible order, after obtaining approval.”

An evaluation should verify the external state. If an agent submits a refund, check the database record. If it sends an email, verify the message delivery. Benchmarks like SWE-bench, WebArena, and OSWorld succeed by focusing on these observable changes rather than just the agent's text output.

A flowchart diagram showing how a single task produces different outcomes, each requiring specific verification.

A benchmark score depends on the whole setup#

Performance is not a property of the model alone. It is a property of the entire system: instructions, tools, memory, retry policies, and time limits.

Research like AI Agents That Matter and Agents Are Systems, Not Models highlights that configuration choices dictate results. A system with a generous retry policy will outperform a one-shot system, but it will also consume more resources. When comparing two versions of an agent, document the environment and budget to know what is actually driving the change in performance.

Reliability takes more than one successful attempt#

Success on a single trial is a poor proxy for reliability. An agent with an 80 percent success rate may have a low probability of succeeding on five consecutive tasks.

τ-bench uses repeated-trial evaluation to distinguish between systems that are lucky once and systems that are consistently reliable. When you evaluate, decide whether you are measuring the probability of a single success or the resilience of the agent across repeated, independent attempts. Use metrics that expose sensitivity to rephrasing or tool outages, as described in Towards a Science of AI Agent Reliability.

The evaluator needs evaluation too#

Using an LLM as a judge is efficient, but it introduces a new source of error. An AI judge can be wrong, just as a programmatic check can be incomplete.

AgentRewardBench and AgentHorizon show that automated judges often struggle to find consequential actions within long, complex trajectories. Your judge must be audited for two specific failures: accepting failed executions and rejecting successful ones. If a judge cannot distinguish between a minor formatting error and a duplicate payment, it is not providing useful data.

Diagnose failures without inventing their causes#

Failed runs are often noisy. You might see a timeout, a retry, and a final error message. The timeout might be a symptom, not the root cause.

AgentRx helps locate critical failure steps by using constraints and recorded violations. Do not guess why an agent failed. Instead, design an intervention: change the specific action you suspect is at fault and re-run the task to observe if the outcome changes.

Simulated users can change the result#

Conversational benchmarks often use a model to simulate a user. However, a simulator that reveals information too early tests a different skill set than a human user who is withholding details. Lost in Simulation and UserProxyBench suggest that simulator-based performance does not always predict human-facing outcomes. If your goal is to support human users, validate your simulation against human behavior.

Build a small evaluation that can teach you something#

Avoid building a massive, opaque test suite. Start with a focused collection of tasks that includes common requests, edge cases, and tool failures.

For every task, define the:

• Initial state
• Intended outcome
• Prohibited side effects
• Required evidence for verification

Run a known solution through the grader to ensure it passes. Then, run plausible near-misses to ensure the grader correctly identifies them as failures.

A rule for the next iteration#

Do not treat an evaluation as a static set of numbers. Use your evaluation to make decisions about prompts, models, and tools. When a score changes, inspect the specific samples that caused the shift. If you cannot explain why a score improved, you do not yet have a reliable evaluation.