Evaluate an AI agent: tests, metrics, and benchmarks

To evaluate an AI agent, check the results it achieves, the actions it takes, and the resources it consumes across a representative set of tasks. Repeat the tests in a controlled environment, then compare those measurements with incidents and outcomes observed in production. Public benchmarks help you shortlist candidates; your own evaluation protocol must establish whether an agent fits your use case and constraints.

Define what success means

A convincing answer does not prove that a task is complete. An agent may report an update without saving the file, modify the wrong folder, or reach the correct outcome after taking a prohibited action. Model evaluation therefore needs to be supplemented by an evaluation of the system that takes action.

Write a success specification before running any tests. It should define the initial state, the request, the permissions, the expected outcome, and the stopping conditions. An ambiguous task mostly produces disagreements between evaluators.

Consider a fictional scenario: preparing a follow-up message for an overdue invoice. Success means finding the correct invoice, checking its status, using the authorized recipient, and saving a draft. Sending the message is a failure if the instructions require approval first.

Separate three aspects:

Outcome: Does the final state match the request?
Behavior: Were the constraints and permissions respected?
Service performance: Are the completion time, cost, and correction workload acceptable?

Also define when declining to proceed is the correct response. Asking for clarification when contact details conflict may count as success. Inventing missing information to finish quickly should count against the agent.

Entries in the AI agent directory help identify advertised capabilities. Your success specification turns those capabilities into conditions you can verify.

Build a representative test set

Start with the tasks you actually intend to assign, including their documents, tools, and complications. Artificially clean requests give an overly favorable picture of everyday performance. Collect examples you are authorized to use, remove unnecessary sensitive data, and have someone familiar with the work validate the expected outcomes.

Distribute cases across routine requests, edge cases, and controlled failures. Include missing information, conflicting documents, insufficient permissions, unavailable tools, and requests outside the agent’s scope. For a document-focused agent, also vary the languages, formats, and age of the sources.

Add closely related variants: a different recipient, a changed date, or an almost identical document. These help establish whether the agent uses the information in the current case instead of reproducing a memorized answer.

Give each part of the dataset a distinct purpose:

Set	Purpose	Access during tuning	How it changes
Development	Understand and fix errors	Available for inspection	Expanded frequently
Validation	Choose between configurations	Limited use	Refreshed under controlled conditions
Held-out test	Estimate final performance	Kept separate	Version frozen for comparison
Regression	Preserve established behavior	Available for inspection	Cases added after a fix is validated

Avoid duplicates across sets, including variants of the same document. Splitting by customer, case file, or time period may protect the held-out test better than a random split.

Keep two views of the results: performance weighted by observed usage, and results broken down by category. Rare but serious cases must remain visible even when they contribute little to the overall average.

Specify each case and control the environment

A test case specification should let another person understand why a run passes or fails. Include a stable identifier, category, inputs, reference data, permitted actions, prohibited actions, and final checks.

Specify which variations are acceptable. Two drafts can express the same request correctly using different wording. An invoice identifier or amount, by contrast, requires an exact check.

Keep the expected outcome inaccessible to the agent. Store grading scripts and reference answers outside its workspace, with separate access permissions.

Before each run, restore the files, database, test accounts, and memory to their initial state. Otherwise, a run may benefit from changes left behind by an earlier attempt. Also document the clock settings and external dependencies when they affect the task.

Two environments serve different needs:

A fixed environment makes comparisons easier: archived documents, recorded tool responses, and restored data.
A connected environment exposes real operating conditions: changed pages, network delays, authentication issues, and service outages.

Keep both. The first helps identify the cause of a regression; the second measures robustness when dependencies change or fail. A tool outage still counts as a service failure, but it needs a separate category so you can direct corrective work appropriately.

Measure quality, reliability, cost, and completion time

Success rate is necessary but insufficient. Report it alongside the number of tasks, the number of runs, and the grading rule. A rate calculated only from completed runs hides interruptions and cases that exceed their budgets.

Use a dashboard in which every metric supports a decision:

Dimension	Measure	What it reveals	Limitation to watch
Outcome	Share of tasks completed successfully in full	Practical usefulness	Hides partial success
Safety	Prohibited actions and attempts	Compliance with scope	A low rate can conceal a serious incident
Intervention	Tasks requiring human recovery	Operational workload	Distinguish planned approval from rescue
Cost	Total spending per successful task	Economic viability	Include failures and retries
Completion time	Median and high percentiles	User experience and capacity pressure	An average hides stalled tasks
Stability	Success across repeated runs	Consistency	Depends on the repetition protocol

Cost per successful task divides all spending for the batch by the number of tasks that succeed under your definition. Include model calls, paid tools, and relevant infrastructure. Report evaluation costs separately from operating costs, and measure the human time required as well.

Distinguish time to the first useful response from time to the verified final state. When reporting a percentile, explain the convention: the 95th percentile is the time at or below which approximately 95% of observations fall.

Partial credit helps diagnose problems, but it must not turn an operation into a pass when a mandatory condition is missing. A prohibited action may remain an automatic failure regardless of the quality of everything else.

Choose and validate your graders

Use programmatic checks when the outcome is observable: whether a file exists, which value was saved, whether software tests pass, who the recipient is, or whether any writes occurred outside the permitted scope. Inspect the final environment instead of relying solely on the agent’s account of what happened.

For a summary or explanation, an editorial rubric can assess accuracy, coverage, supporting evidence, and adherence to the request. Each criterion should describe observable behavior. A general impression of fluency favors polished answers even when they are incomplete.

Anthropic’s Demystifying evals for AI agents distinguishes programmatic graders, model-based judges, and human evaluators. It highlights the need to calibrate automated judges against human assessments.

For calibration, have people annotate a sample without revealing which configuration produced each result. Examine disagreements, clarify the rubric, and then measure false positives: failures that the judge accepts as successes. Check false negatives too, since they may lead you to discard a useful configuration.

An automated judge must treat the outputs it evaluates as untrusted data. An answer under evaluation may contain a prompt injection attempt aimed at the grader.

Finally, test the tests themselves. A draft addressed to the wrong recipient must fail, while an acceptable rewording must pass. A defective grader can make a less reliable configuration look like the better choice.

Repeat runs and make fair comparisons

An agent can succeed on a task and fail when the same task is run again. Repeat tests after restoring the initial state, using a repetition count set before you inspect the results. A low temperature alone does not guarantee identical execution.

Distinguish first-attempt success, success after retries, and consistency across repeated runs. pass@k measures the possibility of obtaining at least one success across multiple attempts; it does not describe the reliability of a normal individual run. Always state whether you are counting observed attempts or using an estimator.

Retries should reflect how you intend to operate the system. If an operator must identify the correct answer among several proposals, that person’s work and the available validation methods belong in the protocol.

Compare configurations on the same cases, with the same time, cost, and action limits. Alternate their order when service load may vary. Examine which tasks each configuration gains or loses, rather than looking only at the overall difference.

Report uncertainty in a way that fits the sample. If you use bootstrapping, resample tasks while keeping their repeated runs grouped together. Variants from the same case file are not necessarily independent observations.

If the difference remains uncertain, report that the experiment does not distinguish the candidates. Expand coverage of situations that matter to the decision instead of repeatedly testing until a favorable result appears.

Read public benchmarks without overgeneralizing

A benchmark measures a system within a specific setup: model, instructions, tools, environment, and budget. Two entries using the same model may represent different agents. Understanding model leaderboards explains how to interpret these results; the benchmark comparison tool provides another starting point.

Several benchmark families can help guide your shortlist:

Benchmark	Area evaluated	Outcome assessed	Limitation for your use case
SWE-bench	Resolving software issues in repositories	Whether the patch passes the protocol’s tests	Does not cover all software maintenance
Terminal-Bench	Tasks performed in a terminal	Success in the specified environment	Depends on the tools and budgets provided
OSWorld	Using applications and a computer	Final state assessed in the environment	Sensitive to applications and interfaces
τ-bench	Interaction among an agent, tools, and a user	Completion under the scenario’s rules	Also depends on the user simulation

Consult the official protocols for SWE-bench, Terminal-Bench, and OSWorld to identify the edition, tasks, and grading procedures.

Benchmark corrections can change scores. As of October 24, 2026, the official τ-bench release notes explicitly state that version 2.0 results are not comparable with results from the 1.x branch.

Archive the exact reference for the protocol you use. Also check whether the tasks are public, whether execution trajectories are available, and whether the result comes from the agent’s authors or an independent evaluation.

Selecting the best run, repeatedly tuning against public tasks, and possible inclusion of those tasks in training all limit what you can infer. A leaderboard helps you decide what to test; it does not provide a success rate for your own business tasks.

Test permissions and unintended effects

Evaluate the actions the agent can actually access, including those exposed through an MCP server. Checking only the final answer will not reveal an unauthorized read or a message sent during execution.

Build plausible adversarial scenarios: a document containing a false instruction, a request that exceeds permissions, a misleading tool result, or an error that encourages bypassing approval. Define the expected behavior: ignore the content, request confirmation, refuse, or escalate to an operator.

Assess model behavior separately from the effectiveness of technical controls. If the agent attempts a prohibited deletion but the environment blocks it, the protection works; the behavior still needs correction. This distinction prevents you from crediting the model for a safeguard enforced elsewhere.

Use dedicated accounts, test recipients, and reversible operations. The guide to securing an AI agent explains containment measures.

Set the conditions that automatically block deployment before running the tests. A data leak must not disappear into an average score for writing quality. Record coverage limits too: passing known adversarial scenarios does not establish the absence of unknown vulnerabilities.

Turn production traces into evidence

Evaluation outside production controls selected situations. Production reveals new requests, changes to documents, and failing dependencies. Both should feed the same measurement loop.

Associate each task with a trace that lets you recover the configuration, tool calls, errors, human approvals, costs, and verified outcome. Record the events needed for diagnosis without routinely collecting all underlying content.

Langfuse describes trace evaluation and dataset experiments in its official documentation. This kind of instrumentation makes comparisons easier, but it does not define your business criteria for you.

Sample apparent successes as well as failures. The absence of a complaint does not prove that a document was processed correctly. Compare user feedback with independent checks.

Monitor results by task type, language, tool, and configuration. A stable average can conceal deteriorating performance in a smaller category. Also maintain immediate operational alerts for exceeded budgets and prohibited actions.

Every confirmed incident should produce a redacted, reproducible regression case. At the same time, refresh a separate dataset of recent cases to track changes in usage. The guide to monitoring an agent in production covers traces and alerts in more detail.

Organize the deployment decision

Before an evaluation campaign, establish the protocol and assign responsibilities. A reproducible process can follow this sequence:

Define the scope and have the success criteria approved.
Version the data, agent, tools, and graders.
Measure a baseline: the current process, fixed automation, or existing configuration.
Run the candidates on the same tasks and analyze regressions.
Decide using thresholds established before seeing the results.
Roll out gradually, with monitoring and a way to restore the previous configuration.

Keep a report that separates measured results, uncertainty, and situations the tests did not cover. Assign an owner to each significant failure according to its source: data, tools, instructions, permissions, or the model. Without that assignment, adjustments may simply move the problem elsewhere.

If you compare the old and new configurations in parallel on real requests, prevent the new configuration from producing external effects. A business operation must not be performed twice for the sake of measurement.

Before expanding the rollout, check that:

Outcomes are verified independently of the agent’s claims.
Failures, retries, and human interventions are included in the assessment.
Permissions and serious incidents have separate criteria.
Comparisons use identical, versioned protocols.
Traces support investigation and restoration of the previous configuration.

The deployment decision can then be revisited using evidence: completed tasks, respected constraints, sustainable costs, and documented limits.

Frequently asked questions

How many tests do you need to evaluate an AI agent?

There is no universal test set size. Start by covering the situations that could change your deployment decision, then expand the set with incidents and requests observed in practice. A small set of clearly specified cases is more useful than a large collection with ambiguous outcomes.

Can you compare agents that use different models?

Yes, if you compare the complete systems for the same use case under the same constraints. This can help you choose a solution, but you cannot attribute the difference to the model alone. To isolate the model's contribution, keep the tools, instructions, and budgets unchanged.

Can a model judge replace human evaluators?

After calibration, it can handle some repetitive checks. Keep human review for disagreements, sensitive cases, and a sample of automated decisions. A judge that appears consistent may share the errors of the system it evaluates.

Is a strong public benchmark score enough to deploy an agent?

No. The benchmark's tasks, permissions, and data may differ substantially from your own operations. Use the score to shortlist candidates. Base deployment on your tests and a monitored pilot.

Do you need to retain every conversation to measure quality?

You can often retain structured events, verification results, and redacted excerpts instead. Define what you need for diagnosis before collecting complete conversations. Evaluation data also needs a defined retention period.

Evaluating an AI agent: test sets, metrics, public benchmarks, and production monitoring