How to evaluate an AI agent: benchmarks and in-house tests

Evaluating an AI agent means checking that it actually completes a task, consistently, not just that it produces a convincing answer. Public benchmarks give you a first reference point, but the real decision rests on your own tests, built from your own use cases.

Why an agent is not evaluated like a chatbot

A classic benchmark asks a question and compares the answer with a reference solution. An AI agent chains actions instead: it looks something up, calls tools, edits a file, sends a message. Agent evaluation therefore has to look at several things at once:

• the outcome: is the final state right (ticket closed, invoice created, test passing)?
• the path: did the agent use the right tools, with no costly detour or risky action?
• consistency: does it succeed every time, or only now and then?
• cost and time: how many tokens and minutes did it take?

Two agents can reach the same result by very different routes, and the same agent can pass and then fail an identical task. That is what makes the job harder than grading a quiz.

What public benchmarks tell you, and what they do not

Public benchmarks act as a shared language. The ones most often quoted for agents:

• SWE-bench, created at Princeton in October 2023, asks the model to fix real issues taken from open-source Python projects; its Verified subset holds 500 tasks re-checked by humans.
• Terminal-Bench, launched on May 19, 2025 by Stanford and the Laude Institute, tests command-line work; version 2.0 came out in November 2025.
• GPQA tests graduate-level scientific reasoning, which is useful to judge the model’s “brain” rather than the full agent.

Our model ranking combines tests of this kind. As of October 22, 2026, the index published by Epoch AI is the average of seven tests, SWE-bench and Terminal-Bench among them. Here is the top of the table:

Rank	Model	Organization	Index
1	Claude Sonnet 5.5	Anthropic	95.9
2	Claude Opus 5.5	Anthropic	95.9
3	Gemini 4 Argon	Google DeepMind	95.3
4	GPT-6 Astra	OpenAI	94.7
5	Claude Opus 5	Anthropic	92.7

And the gap between a few flagship models on that same index:

Capability index (out of 100)

• Claude Sonnet 5.595.9
• Gemini 4 Argon95.3
• GPT-6 Astra94.7
• Claude Haiku 5.584
• GPT-6 Luna80.3

The top five sit within a little over three points of each other. At that level, the difference comes less from the model than from what you build around it: prompt, tools, memory, guardrails. Our methodology explains how these scores are collected and displayed.

Keep three limits in mind. A benchmark measures one precise situation, often coding. Its tasks may have leaked into training data. And the score depends as much on the execution environment as on the model: two teams testing the same model with different tools will not get the same number.

Building your own tests: the vocabulary you need

In its guide “Demystifying evals for AI agents”, published on January 9, 2026 and checked on October 22, 2026, Anthropic sets out a vocabulary that has become standard:

• Task: a single test with inputs and a success criterion.
• Trial: one attempt. You run several per task, because results vary from run to run.
• Grader: the logic that scores one aspect of the work. A task can have several.
• Transcript: the full record of a trial, including tool calls and intermediate steps.
• Outcome: the final state of the environment when the trial ends.

The same guide separates two measures. The first is the probability of succeeding at least once over several trials, which rises as you add attempts. The second is the probability of succeeding on every trial, which falls instead. The first tells you what the agent can do; the second, what you can leave it to do unsupervised.

A five-step method

1. Start from real failures. Anthropic’s guide considers 20 to 50 simple tasks drawn from real failures a great starting point. List what you already check by hand and what users report.

2. Write an unambiguous success criterion. A good test is one where two experts reading the same result would agree. “The answer is helpful” is not enough; “the refund is recorded for the right amount and the customer gets a confirmation message” works.

3. Balance your cases. Add tasks where the agent should decline, ask for clarification or hand over to a person. Otherwise you end up tuning an agent that always acts, even when it should not.

4. Keep two families of tests apart. Capability tests target what the agent cannot yet do well: a low pass rate at first is normal. Regression tests check that it still does what it used to: they should stay close to 100%.

5. Rerun on every change. New prompt, new tool, new model: everything goes through the test suite. Work on instructions, covered in our prompt engineering guide, then becomes measurable instead of a matter of gut feeling.

Choosing your graders

Anthropic’s guide describes three types of graders, best combined rather than pitted against each other:

Grader	Strengths	Weaknesses	Typical use
Code	Fast, cheap, objective, reproducible	Rigid when faced with a valid but unexpected solution	Check that a file exists or a test passes
Model	Flexible, handles open-ended answers and nuance	Non-deterministic, costly, needs calibration	Judge the tone or completeness of a reply
Human	Gold standard, expert judgment	Slow and expensive at scale	Review a sample, calibrate the model grader

The model grader has a name: LLM as a judge. The founding paper by Zheng and co-authors (June 2023) reports over 80% agreement between GPT-4 and human preferences, but also describes biases: favoring the first answer, the longest one, or one from its own model. Hence three habits: swap the order of the answers being compared, ask for a justification before the score, and regularly check the judge against a sample scored by people.

Check yourself Your agent drafts replies to customers and updates their record in a CRM. Which grader for each part?

A script for the CRM update, since the final state can be checked objectively (does the right field hold the right value?). A model grader, calibrated on human scores, for the quality and tone of the message. And a human review of a sample at regular intervals.

Read the transcripts, not just the score

An overall score hides what matters most. When a task fails, the transcript tells you whether the agent really made a mistake or your grader rejected a valid solution. In the same guide, Anthropic describes how a score on the CORE-Bench benchmark went from 42% to 95% once flaws in the evaluation itself, not in the model, were fixed.

The lesson applies to your in-house tests too: a sudden jump in failures deserves a look at the traces before any conclusion. Typical grader bugs include an overly strict format, a wrong reference solution, or an environment that is not properly reset between trials.

Testing safety and judgment

A serious evaluation does not only measure success. It also looks for what can go wrong. Red teaming means attacking the agent on purpose: booby-trapped documents, ambiguous requests, attempts to make it leak data. Add tasks where the right move is to stop and bring in a human in the loop before an irreversible action. Our guide to AI agent security walks through these scenarios.

Tooling: from framework to production

You do not need to code everything yourself. Several frameworks ship evaluation building blocks: Pydantic AI includes Pydantic Evals to score answers and tool-call sequences, and Google ADK offers an evaluation framework with user simulation and custom metrics. Our AI agent frameworks comparison helps you choose.

In production, LLM observability takes over: every call is traced (model, tokens used, tools called, duration), which lets you spot drift and turn real failures into new test tasks. The OpenTelemetry conventions for generative AI, with attributes such as gen_ai.request.model or gen_ai.usage.input_tokens, aim to standardize these traces.

To place evaluation within the whole project, from defining the task to deployment, see our guide on how to build an AI agent.