Automatic AI Agent Evaluation: What to Check with Code, Reference Data, and LLM Judges

An AI agent evaluation should check whether the agent completes the task, uses tools correctly, respects boundaries, and recovers from failure. The strongest approach combines automated code checks, reference data, and LLM judges.

These serve different purposes:

• Code checks verify measurable results, such as whether a file exists or an action required approval.
• Reference data supplies expected answers, source documents, rules, and examples.
• LLM judges assess qualities that require interpretation, such as clarity or whether a clarification question was appropriate.

Both code checks and LLM judging can run automatically. The choice depends on what you need to measure.

Evaluation area	Automated code checks	LLM judge	Data and evidence needed
Task success	Verify the resulting state: tests pass, the correct record exists, or the requested file was updated.	Assess whether an open-ended result satisfies the user’s goal and constraints.	Task description, acceptance criteria, and actual results. Check the outcome independently of the agent’s success message.
Accuracy	Compare exact answers, recalculate numbers, and validate references where possible.	Compare claims with supplied sources and flag unsupported statements.	Verified answers, calculations, and authoritative source documents. The judge’s own knowledge is insufficient evidence.
Instruction following	Check required fields, output format, length limits, and explicit action restrictions.	Assess nuanced requirements, such as preserving the author’s tone or staying within scope.	Original instructions, user preferences, final response, and action logs.
Planning and execution	Detect repeated calls, loops, skipped prerequisites, and budget overruns.	Assess whether observable steps were relevant, sensibly ordered, and sufficient.	Task requirements and execution traces. Allow different valid approaches rather than requiring one exact sequence.
Tool use	Validate tool names, input schemas, argument values, errors, and resulting changes.	Assess whether the selected tool and its use were appropriate for the task.	Tool definitions, inputs, outputs, and environment state before and after execution.
Context handling	Check whether facts and constraints introduced earlier remain reflected in later actions.	Assess whether the agent applies relevant context and handles updated instructions correctly.	Long conversations, handoff records, planted facts, and revised requirements.
Uncertainty	Check whether the agent asks required questions before acting in predefined ambiguous cases.	Assess whether clarification was necessary, assumptions were reasonable, and confidence matched the evidence.	Ambiguous requests, missing-information scenarios, and criteria for when clarification is required.
Error recovery	Simulate timeouts, unavailable tools, and partial failures. Check recovery and duplicate actions.	Assess whether the recovery approach was sensible and unfinished work was explained accurately.	Controlled failure scenarios, retry logs, and resulting state. Include actions that may succeed before a timeout occurs.
Safety and permissions	Detect forbidden actions, missing approvals, access violations, and known sensitive-data leaks.	Assess contextual policy compliance and whether the agent recognized a consequential decision.	Explicit policies, permission boundaries, approval records, and action logs. Critical violations need direct checks wherever possible.
Prompt injection resistance	Insert attack instructions into external content and check for unauthorized actions or disclosures.	Assess whether malicious content subtly redirected the answer or influenced decisions.	Adversarial documents, websites, and tool responses, paired with expected authorized behavior.
Efficiency	Measure latency, token usage, tool calls, retries, and total cost against a budget.	Assess whether additional work contributed useful value.	Usage logs and task difficulty. Compare efficiency alongside success so that incomplete work does not appear better merely because it was cheaper.
Communication	Check required information, formatting, and consistency with recorded execution results.	Assess clarity, relevance, and honest reporting of uncertainty, limitations, and unfinished work.	User request, final response, and execution evidence. Look for unsupported claims that work was completed.

What this looks like in practice#

Suppose a user asks an agent to schedule a meeting.

A code check can confirm that the event exists, includes the correct attendees, and falls within everyone’s available hours. Reference data supplies calendars, time zones, and scheduling constraints. An LLM judge can assess whether the agent handled an ambiguous request appropriately and clearly explained the result.

The same task needs several checks because a calendar event can exist while still being scheduled incorrectly.

How to score the results#

For each criterion, record pass, partial, or fail, together with the supporting evidence and failure severity. Give LLM judges specific definitions of these ratings and ask them to cite relevant evidence.

Test routine tasks, ambiguous requests, edge cases, and adversarial inputs. Repeat important cases to measure consistency, and regularly compare LLM ratings with human reviews.

Use code wherever correctness is objectively checkable. Use an LLM where interpretation is necessary. Treat critical safety failures as release blockers, regardless of the average score.

This approach follows the combination of code-based, model-based, and human grading described in Anthropic’s guide to agent evaluation.