LLM benchmarks: how to read model rankings
An LLM benchmark measures a specific skill under specific conditions: it does not tell you whether a model is “intelligent” or whether it will suit your needs. To read a ranking, check four things: what the test measures, whether it is already saturated, whether the gap between models exceeds the margin of error, and how much each additional point costs. The figures in this article come from Epoch AI data and OpenRouter pricing, recorded on October 24, 2026.
What a benchmark measures—and what it leaves out
A benchmark is a set of questions or tasks with known correct answers, given to each model under the same conditions. The score is the proportion completed successfully.
All benchmarks share three limitations:
- A narrow skill. A competition math test tells you nothing about a model’s ability to edit a code repository or follow a business procedure through 40 steps.
- Fixed conditions. The number of attempts, the amount of reasoning allowed, and access to tools all matter. Changing just one of these settings can shift a score by several points.
- Contamination. Once questions circulate online, they can end up in training data. A model may then recall the answer instead of working it out. This is why several recent benchmarks, including FrontierMath, keep some of their questions private.
The benchmarks you are most likely to encounter
The table below lists the benchmarks shown on our model profiles, along with the number of models scored in Epoch AI’s data as of October 24, 2026.
| Benchmark | What it measures | Models scored | Highest recorded score |
|---|---|---|---|
| GPQA Diamond | PhD-level science questions, multiple choice (4 answers) | 124 | 96.4% |
| OTIS Mock AIME 2024-2025 | Competition math problems | 124 | 100% |
| FrontierMath (levels 1-3) | Research-level math problems, private questions | 85 | 93.7% |
| SimpleQA Verified | Accuracy on short factual questions | 80 | 76% |
| ARC-AGI-1 | Visual puzzles testing abstract reasoning | 73 | 98.5% |
| ARC-AGI-2 | A harder version of ARC-AGI | 73 | 95% |
| Terminal-Bench | Real tasks in a terminal | 37 | 84.7% |
| SWE-bench Verified | Fixing real GitHub issues in Python projects | 28 | 83.5% |
| Aider Polyglot | Coding exercises in six languages | 25 | 88% |
| OSWorld | Computer use involving applications and files | 6 | 72.1% |
Sources: GPQA, SWE-bench, Terminal-Bench, OSWorld, Aider, ARC-AGI. The LLM benchmark tool in the Lab lets you sort and filter these scores by model.
There are two things to keep in mind when reading this table. First, the benchmarks closest to real agent work—SWE-bench Verified, Terminal-Bench, and OSWorld—also have the fewest models scored. Comparisons on these tests therefore cover a smaller selection of models. Second, ARC-AGI serves a distinct purpose: it was designed to measure the ability to solve unfamiliar problems rather than recall knowledge. For a detailed explanation, see ARC-AGI: the benchmark, its versions, and its scores.
First pitfall: saturation
A benchmark is saturated when most recent models score close to its maximum. It can no longer distinguish meaningfully between them.
In the data recorded:
- On GPQA Diamond, 77 of the 124 models scored exceed 80%, and the median score is 85.3%.
- On OTIS Mock AIME, 79 of the 124 models exceed 80%, and several reach 100%.
- By contrast, on SimpleQA Verified, none of the 80 models scored reaches 80%. The highest score is 76%.
In practice, a two-point gap on GPQA Diamond between leading models tells you very little. The same gap on a benchmark that is far from saturated is more informative. Before comparing individual scores, look at how models perform across the test as a whole.
Second pitfall: ignoring the margin of error
Every score is an estimate with some uncertainty. A ranking sorted to a tenth of a point often conceals ties.
The Epoch Capabilities Index (ECI), which combines many benchmarks into a single scale of “general capability,” publishes an uncertainty interval. As of October 24, 2026, its top three models are:
| Model | ECI | Interval |
|---|---|---|
| Claude Opus 5.5 | 168.8 | 165.3 – 174.0 |
| GPT-6 Astra | 168.8 | 165.5 – 174.5 |
| GPT-6.1 Sol | 168.4 | 165.0 – 174.4 |
The intervals overlap almost completely: these three models are tied on this index. A headline claiming that “model X dethrones model Y” on the strength of such a small gap goes beyond what the data supports.
A practical rule: if the difference between two models is smaller than the width of their intervals, treat them as equivalent. Choose between them using other factors, such as price, speed, context window, or license.
Third pitfall: overlooking price
Models with equivalent capabilities can differ in price by a factor of five or more. In the OpenRouter pricing recorded on October 24, 2026:
- GPT-6 Astra costs $10 per million input tokens and $50 per million output tokens.
- GPT-6.1 Sol, at the same ECI level, costs $2 and $10.
- Claude Haiku 5.5, with an ECI of 159.3, costs $0.10 and $0.50.
For an agent that calls the model dozens of times per task, that difference has a substantial effect on the monthly bill. Each model profile includes a price versus score chart showing how it compares with other models. The LLM pricing calculator turns those rates into a monthly estimate based on your usage. Prices change, so use the model profile, updated with each data collection, for current rates rather than relying on the figures in this article.
Fourth pitfall: relying on the provider’s score
Providers publish their own scores when launching a model. These are useful, but they are still launch announcements:
- The provider selects benchmarks where its model performs well.
- It chooses the conditions, including the number of attempts, maximum reasoning effort, and available tools.
- It often compares its model with older versions of competing models.
Independent evaluations—from Epoch AI, the ARC Prize, or leaderboards maintained by benchmark authors—provide a more consistent basis for comparison. Check the date, too: a score published six months ago often compares models that have already been replaced.
Agent benchmarks: closer to real work, but still young
When choosing a model for an AI agent, tests involving multiple steps and tools are the most informative:
- SWE-bench Verified: the model receives a real issue and a code repository, then must produce a patch that passes the tests.
- Terminal-Bench: tasks to complete in a terminal, such as installation, configuration, and debugging.
- OSWorld: tasks on a real operating system, using a mouse, keyboard, and applications.
Their limitation is that the score depends as much on the agent—its tools, execution loop, and instructions—as on the model. The same model can gain or lose several points depending on the tools and setup around it. This is why our AI agent profiles assess the agents themselves, covering activity, licensing, compatible models, and risks, alongside our model profiles.
A method built around five questions
Before drawing a conclusion from a ranking:
- What does this test measure, and how closely does it match my task?
- Is it saturated? What is the median model score?
- Does the gap exceed the margin of error?
- Who ran the evaluation, under what conditions, and when?
- What does the difference cost? Is the gain worth the additional expense at my usage volume?
If the choice is still close, your own test remains the most reliable guide: take around twenty real examples of your task, run them through the two or three shortlisted models, and score the results using a rubric written in advance. The guide Choosing an LLM for your agent explains this method in detail, while The cheapest LLM for each use case applies the price criterion.
Key takeaways
- A benchmark measures a narrow skill under fixed conditions, not general intelligence.
- Saturated tests such as GPQA Diamond and AIME no longer distinguish between leading models.
- A gap smaller than the margin of error is a tie: in October 2026, this applies to the top three models in the ECI.
- Models with equivalent capabilities can differ in price by a factor of five: always compare the cost per task.
- For the broader question of how to measure general intelligence, see What is AGI?.
Frequently asked questions
What is the best benchmark for comparing LLMs?
No single benchmark is enough. A composite measure such as the Epoch Capabilities Index gives you a broad overview; a benchmark that matches your task, such as SWE-bench Verified for coding, OSWorld for computer use, or SimpleQA Verified for factual accuracy, tells you more about your needs. Ideally, run a small test using your own examples.
Why do different rankings show different results?
They measure different things under different conditions: the questions selected, the number of attempts allowed, reasoning settings, available tools, and the exact model version. A difference of a few points between rankings is normal and often falls within the margin of error.
Can you trust scores published by model providers?
They are useful, but treat them as launch announcements: providers choose the benchmarks where their models perform well and set the testing conditions. For comparisons, favor independent evaluations such as those from Epoch AI or the ARC Prize, and check their dates.
Does a score of 90% mean the model gets one in ten answers wrong?
Only on that benchmark's questions, under its testing conditions. The error rate on your own tasks can be very different: lower for simple tasks, much higher for subjects the benchmark does not cover or for long tasks involving multiple steps.