AI benchmarks explained, and the traps to avoid

An AI benchmark is a standardised test that scores models on a specific skill: scientific reasoning, maths, coding or agent tasks. Read well, these tests help you compare models. Read badly, they mislead, because each one has blind spots.

What is a benchmark for?

A benchmark is a fixed set of questions or tasks with a fixed scoring rule. Several models take it under identical conditions, which makes their scores comparable. Think of it as a common exam: useful for ranking candidates, but it never captures everything they can do.

Our AI model ranking is built on data from the research institute Epoch AI, published under a CC BY 4.0 licence. Here is the top of the table as of 22 October 2026.

Rank	Model	Organization	Index
1	Claude Sonnet 5.5	Anthropic	95.9
2	Claude Opus 5.5	Anthropic	95.9
3	Gemini 4 Argon	Google DeepMind	95.3
4	GPT-6 Astra	OpenAI	94.7
5	Claude Opus 5	Anthropic	92.7
6	Claude Fable 5.1	Anthropic	92.6
7	GPT-6.1 Sol	OpenAI	92.1
8	Claude Fable 5	Anthropic	90.8
9	GPT-6 Sol	OpenAI	89.2
10	GPT-5.5	OpenAI	89

The 7 tests behind the ranking

The ranking uses a basket of 7 benchmarks. The descriptions below come from Epoch AI’s official pages, accessed on 22 October 2026 (epoch.ai/benchmarks).

Benchmark	What it measures	Size	Public questions?
GPQA Diamond	PhD-level science questions, 4 options each	198 questions	Not published in plain text
OTIS Mock AIME 2024-2025	Competition maths problems, answers from 000 to 999	45 problems	Not specified by the publisher
FrontierMath	Research-level maths, unpublished problems	Several hundred	Mostly not
SWE-bench Verified	Fixing real bugs in Python projects	500 tasks	Yes
FrontierCode	Writing a patch good enough to be merged	150 tasks	Public issues
Terminal-Bench 2.0	Completing tasks in a terminal	Not specified by the publisher	Not specified by the publisher
APEX-Agents	Long professional tasks across several apps	480 tasks	Not specified by the publisher

A few useful details:

• GPQA Diamond (Epoch AI page): 198 multiple-choice questions in biology, physics and chemistry. Random guessing scores 25%, which sets the floor.
• OTIS Mock AIME (Epoch AI page): 45 problems written by students on an olympiad training programme, harder than school exercises but easier than FrontierMath.
• FrontierMath (Epoch AI page): unpublished problems written by mathematicians across four tiers, plus a set of open research problems.
• SWE-bench Verified: 500 human-reviewed tasks drawn from real issues in open-source Python projects.
• FrontierCode (Epoch AI page): built by Cognition, it judges whether an agent’s patch would be accepted by maintainers, meaning the behaviour is fixed, nothing else breaks, and the project’s style and conventions are respected.
• Terminal-Bench (Epoch AI page): tasks carried out in a terminal, the way a developer or system administrator would.
• APEX-Agents (Epoch AI page): built by Mercor, 480 tasks split evenly between investment banking, management consulting and corporate law. According to Epoch AI, a professional would need 1.8 hours on average for each one.

How is the index calculated?

Raw scores can’t be compared across tests: 60% on FrontierMath is worth far more than 60% on GPQA. Our methodology therefore normalises them test by test. On each benchmark, the best model evaluated gets 100 and the weakest gets 0, with the rest in between. The index is the average of these normalised scores, and it’s only calculated for models scored on at least 4 of the 7 tests.

This has two practical effects. First, the index is relative: when a new model sets a record on a test, everyone else’s normalised score drops, even if their raw results haven’t changed. That’s why an older model can lose points over the months. Second, two models that haven’t taken exactly the same tests are compared on slightly different grounds.

The traps to watch for

Contamination

Models learn from huge amounts of text collected from the web. If a test’s questions and answers are in there, a model can pass by memory rather than reasoning. That’s contamination. The risk is higher for long-standing public tests: SWE-bench and FrontierCode, for instance, are built on public issues from open-source projects. By contrast, FrontierMath keeps most of its problems secret, and the GPQA authors ask people not to publish examples in plain text online and distribute the dataset in a password-protected archive.

Saturation

A test stops being useful once every strong model is close to the maximum. According to Epoch AI, the best GPQA Diamond score went from 35.7% (GPT-4) in the first quarter of 2023 to 76.8% (o1) at the end of 2024, then 92.6% (Gemini 3 Pro) at the end of 2025 and 96.4% (Gemini 4 Argon) in the third quarter of 2026. At that level, the gaps between leaders become thin and the test no longer separates them well. That’s why tougher benchmarks keep appearing.

Noise in small tests

With 45 problems, each right or wrong answer moves the score by about 2.2 points. With 198 questions, by about half a point. So a one or two-point gap between two models may just be chance. Epoch AI repeats runs to smooth out this noise, but it never disappears.

Testing conditions

A score depends on how the model is queried: the prompt, the number of attempts, the settings and how much thinking time reasoning models get. Epoch AI says it uses an empty system prompt and the APIs’ default settings. For Terminal-Bench the issue is even sharper: the score depends on the model and agent pairing, because the tool driving the model changes the result. According to Epoch AI, only the best combination is kept for each model, among agents such as Claude Code, Codex CLI, Terminus and Goose.

The judge model

Some tests can’t be marked automatically. APEX-Agents scores each task against an expert rubric applied by another model, which is the LLM-as-a-judge approach. It works at scale, but it inherits the marking model’s biases.

The gap with your own use

None of the 7 tests measures the quality of an email, a summary or a translation. A model that ranks very high in maths can be less pleasant to use day to day than a more modest one. For agents, our guide on evaluating an AI agent suggests a method based on your own tasks.

What benchmarks say about price

Benchmarks also track the cost of performance. According to Epoch AI, the cheapest model reaching 70% on GPQA Diamond cost $26.25 per million tokens at the end of 2024, with o1. By the third quarter of 2026, that figure had fallen to $0.055, with Qwen3.7 Flash. A level once limited to the most expensive models is now available for a fraction of the price, though that doesn’t make the cheap model a match for a frontier model at everything else.

Five habits for reading a ranking

1. Check which tests count. An overall index mixes skills. If you code, the SWE-bench and FrontierCode columns matter more to you than GPQA.
2. Don’t over-read small gaps. One or two points at the top may be noise.
3. Check the date. Rankings age fast, and a model released last week can reshuffle everything.
4. Compare like with like. Mixing scores from different sources skews the comparison.
5. Test for yourself. The score that matters most is the one a model gets on your tasks.

Check yourself A model scores 2 points higher than another on a 45-problem test. Is it clearly better?

No. On 45 problems, one extra correct answer is worth about 2.2 points, so the gap may come down to the luck of a single run. Look at all the tests and at clear-cut gaps instead.

Rankings also shape the business landscape, as they affect how model makers are perceived, as our guide who builds AI? shows. They help place European AI against American and Chinese models, and compare open and closed models, the subject of our article on open-source vs proprietary AI. Finally, evaluating the most powerful models is part of the obligations set by the EU regulation covered in our AI Act summary.