Benchmarks: how AI models get compared
A benchmark is a standardised test: the same set of questions or tasks, scored the same way, given to several models so their results can be compared. It is where most of the scores and AI rankings you see online come from.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|
What it means
In everyday English a benchmark is a reference point. In AI, it is a fixed set of tests with a scoring rule: multiple-choice questions, coding problems to solve, tasks to complete on a computer, and so on. Because every model sits the same exam, their scores can be compared.
A concrete example
One of the most cited benchmarks of recent years is MMLU, introduced in 2020 in the paper “Measuring Massive Multitask Language Understanding” (Hendrycks et al.). It covers 57 subjects, from elementary maths to law and history, as multiple-choice questions. Other benchmarks target more practical skills: SWE-bench checks whether a model can fix real bugs in software projects, while GPQA asks very hard science questions designed to resist a quick web search.
Why it matters
Without benchmarks, every model maker could claim to have “the best” model with no way to check. Benchmarks provide a shared language for tracking progress, spotting a model’s strengths and choosing the right one for a task. For AI agents, dedicated tests measure the ability to chain actions together, which is the field of agent evaluation.
On this site, the methodology page explains where the displayed scores come from and how they are combined, and the AI ranking puts them in context every week.
Traps to watch for
- Contamination. If a benchmark’s questions end up in a model’s training data, the model may “know the answers” without genuinely reasoning.
- Saturation. When the top models all score close to the maximum, the test can no longer tell them apart, and newer, harder benchmarks take over.
- Test conditions. A score achieved with several attempts, tools or a tuned prompt is not comparable with a single-attempt score.
- The gap with real use. A benchmark measures a narrow task. Your day-to-day experience also depends on speed, price, style and reliability on your own requests.
Related terms
When an answer cannot be scored automatically, some protocols ask another model to grade it. That is the idea behind LLM-as-a-judge: handy, but imperfect.
Related terms
- Agent evaluation · Methods for measuring whether an AI agent really completes a task end to end, reliably, safely and at an acceptable cost.
- SWE-bench · A benchmark that asks an AI to resolve real GitHub issues from Python projects, with each fix checked against the project's own tests.
- GPQA · A benchmark of PhD-level multiple-choice questions in biology, physics and chemistry, designed so a web search does not reveal the answer.
- Terminal-Bench · A benchmark that measures whether an AI agent can complete hard, realistic tasks on its own inside a command-line terminal.
- LLM-as-a-judge · An evaluation method in which a language model scores or compares another model's answers against a rubric, standing in for a human grader.
Try it
Full glossary
Frequently asked questions
Does a higher benchmark score mean a better AI for me?
Not necessarily. A benchmark measures one skill under specific conditions. If you mostly draft emails, a top score in advanced maths tells you little, so look for benchmarks close to your own tasks.
Who creates benchmarks?
Mostly academic researchers, AI labs and independent organisations, who usually publish the questions and the scoring method. Model makers then choose which benchmarks to highlight in their announcements.
Why do reported scores sometimes differ between sources?
Test conditions vary: prompt wording, number of attempts allowed, access to tools, model settings. An evaluation run by an independent organisation with a public protocol is more comparable than a single figure in a press release.
How does this site use benchmarks?
The site uses scores published by named public sources and explains its approach on its methodology page. The weekly ranking builds on that data without claiming to measure how each user experiences a model.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.