AGI benchmarks: measuring progress and their limits

Benchmarks measure how well AI systems perform on defined tasks; they do not directly measure how far those systems are from AGI. To interpret progress, you need to examine the tasks, evaluation conditions, resources used, and potential contamination. A credible assessment combines several tests with an analysis of generalization, errors, and cost.

Define what you want to measure

A benchmark brings together a set of tasks, an evaluation procedure, and a scoring rule. Its score answers a specific question: how well does the system perform on these tasks under this configuration? Moving from that observation to a claim about general intelligence requires additional evidence and reasoning.

In particular, distinguish between acquired skills, the breadth of those skills, and the ability to acquire new ones. Answering a scientific question draws on knowledge; discovering the rules of an unfamiliar environment requires adaptation. A system can improve on the first dimension without making comparable progress on the second.

Reliability is another dimension. A correct solution reached after many attempts does not demonstrate the same behavior as repeatable success without assistance. In a practical application, the time required, human interventions, and consequences of failure matter too.

Before reading a leaderboard, identify the claim you want to assess. You might need a better document assistant, a system that can learn a new rule, or an autonomous agent for extended tasks. The AGI and research hub distinguishes these questions, which cannot all be reduced to a single scale.

How definitions of intelligence shape the tests

In On the Measure of Intelligence, published in November 2019, François Chollet proposes measuring intelligence through the efficiency of skill acquisition, accounting for prior knowledge, experience, and generalization difficulty. His critique concerns the tendency to equate mastery of a task with intelligence: more training can improve task performance without, by itself, establishing a greater ability to adapt.

This perspective asks what the system knew before the evaluation and what it has to discover during it. It does not imply that knowledge is useless. It requires separating the contribution of existing knowledge from that of learning in response to something new.

Other frameworks focus on describing observable capabilities. In Levels of AGI for Operationalizing Progress on the Path to AGI, Meredith Ringel Morris and her coauthors distinguish depth of performance from breadth of capabilities, then discuss their relationship with autonomy. Strong performance in a narrow domain is therefore insufficient to establish generality.

These approaches place the emphasis differently: one focuses on the efficiency of skill acquisition, while the other organizes levels of capability. They do not provide a common formula for converting scores into AGI. The page on generalization explores this distinction, which helps explain why researchers choose different kinds of tests.

An evaluation suite should cover complementary capabilities. The following table describes families of tests rather than ranking individual benchmarks: what a result means always depends on the specific evaluation protocol.

Test family	What is observed	Main strength	Limitation to examine
Knowledge questions	Answers to predefined questions	Scoring can be standardized	Difficulty separating knowledge from adaptation
Problems with verifiable solutions	A mathematical, logical, or computational result	Less subjective verification	How representative the selected problems are
Software development tasks	Changes to a program followed by validation	An executable result	Test coverage and quality
Interactive environments	Successive actions and responses to observations	Evaluation of exploration	Dependence on the interface and resource budget
Human preferences	Choices between responses	Measurement of perceived usefulness	Influence of style and evaluators
Learning new rules	Success after demonstrations or exploration	Observation of adaptation	How novelty and prior knowledge are defined

These families do not necessarily need to be combined into one score. An average can hide a significant weakness: an agent that performs well at programming may still struggle to interpret an ambiguous request or recognize that information is missing.

The choice of tasks also reflects an intended use. Academic questions, software repairs, and interactions with a computer represent different activities. Difficulty alone does not guarantee that a task provides useful evidence about general intelligence.

For help reading leaderboards when choosing a model, see LLM benchmarks: how to read the rankings. The question here is broader: what does each result actually allow you to claim about the system’s capabilities?

What ARC-AGI contributes to measurement

The official overview of the ARC-AGI series connects its evaluations to acquiring skills on unfamiliar tasks. It emphasizes the distinction between a specialized skill and the ability to generalize. That focus makes it a reference point for studying adaptation, rather than simply testing an inventory of knowledge.

In a grid transformation task, for example, the system must identify a rule consistent with the demonstrations and apply it to a new input. The relevant difficulty is not just computational: the system must choose a representation and distinguish between competing hypotheses.

Interpretation still depends on prior experience. Solving a task never seen before is not equivalent to encountering an unfamiliar family of problems. Training on related tasks, synthetic data, or specialized search procedures can contribute to the result; their use needs to be documented.

An ARC-AGI score does not, on its own, measure every linguistic, social, physical, or professional capability. Nor is it sufficient to establish an agent’s operational autonomy. The ARC-AGI reference page explains the tasks and their evaluation protocols in more detail.

The useful approach is to connect each success to its mechanism and conditions. Examine whether the system discovers a rule from limited information, how much search it needs, and whether the method remains effective when the presentation or task family changes.

Saturation: when scores become less informative

A benchmark becomes saturated when results approach its ceiling closely enough that differences become difficult to interpret. The remaining failures can then have a disproportionate effect on rankings, even when they reflect ambiguous scoring or an unusual feature of the test set.

Saturation is not evidence of contamination. A research community may learn to solve a class of problems through advances that transfer to other tasks. Conversely, a benchmark can remain difficult even when its contents have appeared in training data. These phenomena need to be examined separately.

A test near its ceiling can still serve as a regression check. It becomes less useful for distinguishing the strongest systems. To recover useful information, evaluators can change the distribution of tasks, demand greater reliability, or compare the resources required to achieve similar success rates.

Creating a harder test is not enough, however. An obscure instruction or disputed scoring rule can increase failures without improving the measurement. Evaluators need to check that the added difficulty corresponds to the capability they intend to assess.

Replacing a benchmark also breaks continuity between scores. A lower score on a new test does not necessarily indicate that the system has regressed. To track progress over time, retain some shared evaluations across periods and explain what the new test makes it possible to observe.

Contamination: exposure, memorization, and optimization

Benchmark contamination compromises the separation between material used for learning and material used for evaluation. It can arise from publicly available web content, answer keys, translations, or paraphrases incorporated into training. It does not necessarily imply an intent to deceive.

Several situations need to be distinguished. The system may have seen the exact question together with its answer, or it may have encountered only a similar question. Developers may also have adjusted their system after observing many results on the same test. These mechanisms do not have identical effects.

Exposure does not automatically establish useful memorization, and a correct answer does not establish exposure. The practical question is whether the score overstates performance on comparable new tasks.

In ConStat, published in May 2024, Jasper Dekoninck, Mark Niklas Müller, and Martin Vechev propose studying this inflation by comparing performance on a benchmark with reference evaluations, relative to other models. Their approach examines performance that fails to transfer to comparable tests, rather than looking only for examples present in a training corpus.

A drop in performance after rephrasing still requires interpretation: the modified version may have changed the difficulty. Different sources of evidence need to be considered together. Without a complete audit, a precise account is more useful than an absolute verdict: explain the checks performed, the data available, and the uncertainties that remain.

Keep tests fresh without sacrificing reproducibility

Public tests allow researchers to inspect errors, reproduce calculations, and challenge scoring decisions. Publishing them also makes it easier for their contents to enter future training runs. Private tests reduce that exposure but restrict independent auditing.

One strategy is to separate functions: use a public set to develop methods, a held-out set to compare systems, and refreshed tasks to check transfer. This separation must also cover answer keys and the detailed feedback shared with participants.

A private test can become a development tool if submissions are repeated and feedback is too detailed. Limiting attempts, keeping a submission history, and reserving a final evaluation reduce this risk. Confidentiality alone therefore does not guarantee independence.

Automatically generated tasks provide fresh material, but volume does not guarantee diversity. A generator may produce superficial variations of the same underlying mechanism. Evaluators need to examine the families of rules, possible strategies, and available shortcuts.

To document these choices, report when the tasks were created, access conditions, duplicate checks, and the policy for refreshing them. A recently created set offers some evidence of novelty; it does not establish that the ideas needed to solve it were absent from the system’s prior experience.

Compare the complete system and its resources

A result attributed to a model may depend on an entire pipeline: instructions, tools, memory, solution search, answer selection, and stopping criteria. If these elements change, you are comparing different systems, even when the underlying model stays the same.

The official SWE-bench website distinguishes several task sets and also provides a view in which models use a shared environment. This distinction illustrates why the choice of tasks should be considered separately from the architecture used to execute them.

The number of attempts deserves particular attention. Success on the first response, the presence of a correct solution among several candidates, and the system’s ability to select that solution are different measurements. An evaluator that knows the answer should not be confused with autonomous selection by the system.

The resource budget should accompany the score: elapsed time, resource consumption, tool calls, and human interventions. Increasing resources can improve success rates. That does not make the improvement meaningless, but it changes its practical significance.

To compare two systems, ask for a curve showing the relationship between resources and performance, rather than a single best result. The guide to evaluating an AI agent applies this reasoning to practical uses, where success that takes too long or costs too much may fail to meet the need.

Look at uncertainty and errors behind the average

An aggregate score depends on the tasks selected. If most come from the same family, their number can create a misleading impression of broad coverage. Closely related variants do not each provide an independent observation of a general capability.

When comparing systems, examine their results on the same tasks and look at where those results differ. Repeat runs when the procedure involves randomness. A small gap may be due to the responses sampled, the selection of problems, or the scoring rules.

The scoring itself also needs scrutiny. A correct answer may be penalized for its format; a software solution may pass an incomplete test suite; a persuasive response may convince a judge despite containing an error. Auditing individual examples can reveal these limitations.

The Holistic Evaluation of Language Models program, presented in November 2022, advocates evaluation across multiple scenarios and dimensions, including calibration, robustness, and efficiency. Its methodological contribution is to expose trade-offs that accuracy alone can hide.

In a report, distinguish success rates, serious errors, appropriate refusals, and abandoned attempts. Include results by category and analyzed examples. An average is useful as a summary; it cannot replace a description of the evaluation’s scope or an examination of failures.

Build a dashboard for progress toward AGI

A lasting evaluation framework should make progress visible without claiming to convert it into a percentage of AGI. It can track the breadth of tasks completed successfully, adaptation to novelty, consistency, autonomy, and the resources required.

In its overview of a cognitive framework published on March 17, 2026, Google DeepMind also proposes grounding measurement in a taxonomy of cognitive capabilities. This is another way to organize evaluation coverage; it too needs to be judged by the tests it helps researchers construct.

For each new publication, look for documented answers to these questions:

Which capability is being evaluated, and which capabilities remain outside the scope?
Are the tasks new to both the system and its developers?
Are the protocol, resources, and stopping criteria comparable?
Does the improvement carry over to other tasks of the same kind?
Are the remaining errors understood, and are the results reproducible?

Evidence becomes stronger when improvements converge across independent tests, with documented resources and verified transfer. Its conclusions still remain limited to the scope observed. When tracking progress toward AGI, the useful result is therefore a profile of capabilities and their uncertainties, rather than an isolated threshold presented as a definitive verdict.

Frequently asked questions

Is there a benchmark that proves an AI has reached AGI?

That judgment first requires a shared definition of AGI and agreed acceptance criteria. A test can establish a capability within its scope; neither its name nor its difficulty automatically gives the result broader significance.

Does a saturated benchmark become useless?

It can still help detect regressions or compare less capable systems. To distinguish the strongest systems, however, evaluations need tasks or constraints that still provide useful information.

Can contamination be detected without access to training data?

Some methods look for clues in model probabilities or differences in performance between original tasks and modified versions. Their interpretation depends on statistical assumptions: they do not provide a comprehensive audit of the training corpus.

Why do two leaderboards produce different rankings?

They may evaluate different tasks, use different resource budgets, or apply different scoring methods. Also check whether each result comes from a model alone, an agent with tools, or a specialized configuration.

Can a better benchmark score predict when AGI will arrive?

A forecast needs an explicit model connecting observed measurements to the capabilities still missing. Extending a score trend without establishing that connection does not produce a reliable timeline.

Measuring progress toward AGI: benchmarks, limits, saturation, and contamination