AGI guide · Beginner · 10 min
ARC-AGI: the test, its versions, and published scores
ARC-AGI tests adaptation to unfamiliar problems. Learn how its three versions work, what published scores show, and where comparisons need greater care.
ARC-AGI is a series of tests that evaluates how well a system discovers rules and adapts to unfamiliar problems from a small number of examples. ARC-AGI-1 and ARC-AGI-2 use colored grids; ARC-AGI-3 introduces interactive environments to explore. Their scores measure specific capabilities, but they do not certify that a system has achieved artificial general intelligence.
What ARC-AGI aims to measure
The starting point is a distinction between having a skill and being able to acquire a new one. A program may master an exercise because its developers supplied many examples or built a specialized method. That success alone does not establish how it will respond to a different kind of challenge.
In On the Measure of Intelligence, published in November 2019, François Chollet proposes defining intelligence in terms of skill acquisition efficiency, while accounting for prior knowledge, experience, and generalization difficulty. ARC, short for Abstraction and Reasoning Corpus, is the experimental framework associated with that proposal.
The test therefore targets a form of fluid intelligence: understanding an unfamiliar situation and developing a solution, rather than recalling specialized knowledge. Here, generalization means moving from situations already observed to a new situation.
This goal does not imply that a system starts with no knowledge. Recognizing objects, their continuity, or symmetry already requires expectations about the world. The question becomes what initial knowledge, additional experience, and resources the system needs to succeed.
For context on how this approach fits into other areas of research, the AGI and research hub brings together the site’s reference pages.
How a grid task works
In the static format, you examine pairs of grids: an input and its corresponding output. You must identify a transformation consistent with the demonstrations, then produce the output for a new input. The rule is not supplied in natural language.
Consider a fictional teaching example that does not reproduce an official task. The demonstrations show a shape surrounded by a frame; the output retains only the shape inside. To solve a new input, you need to understand the frame’s role, identify what it encloses, and construct the expected grid.
Other hypotheses might explain an isolated demonstration, though: retain the dominant color, for example, or remove objects around the edges. The remaining examples help distinguish between these interpretations. Part of the challenge is therefore choosing the right way to represent the problem.
The required answer is an exact grid, with the correct dimensions and cells. A convincing explanation accompanied by an incorrect grid does not count as a correct solution.
The official ARC-AGI-1 repository, accessed October 24, 2026, contains the data and an interface for solving tasks manually. It also describes the file format. Historical protocols require care: the repository mentions three attempts, whereas ARC Prize evaluations use protocols including pass@2. Always check the rules attached to the score being cited.
How the three versions differ
The versions are more than interchangeable difficulty levels of the same exam. Moving to interaction changes what a system must discover and how its success is measured.
| Criterion | ARC-AGI-1 | ARC-AGI-2 | ARC-AGI-3 |
|---|---|---|---|
| Release milestone | 2019 | March 2025 | March 2026 |
| Format | Input and output grids | Input and output grids | Interactive environments |
| Available information | Transformation demonstrations | Transformation demonstrations | Observations obtained through actions |
| Required work | Infer a transformation | Combine more demanding rules | Explore, discover the goal, and act |
| Interpreting the score | Exact solutions under the protocol | Exact solutions under the protocol | Success and action efficiency |
These milestones appear in the ARC-AGI-3 technical report, dated April 22, 2026. The third version launched on March 25, 2026.
ARC-AGI-1 remains useful for studying methods and comparing historical results. However, scores approaching its ceiling make it less effective at distinguishing among the strongest systems.
ARC-AGI-2 retains the transformation problem but places greater demands on composition and contextual interpretation. ARC-AGI-3 adds a separate challenge: choosing which observations to gather. Instead of receiving all the examples upfront, the system must partly organize its own learning experience.
An identical percentage on different versions therefore does not represent an identical capability. Any comparison needs to account for both the scoring metric and the content of the tasks.
ARC-AGI-2: combining and interpreting rules
ARC-AGI-2 was introduced on March 24, 2025. Its official announcement describes tasks designed to test symbolic interpretation, the composition of transformations, and context-dependent rules.
The challenge goes beyond recognizing a pattern. An element may act as a signal to transform another object; a rule may apply only within a particular region. A successful method must select the relevant relationships, rather than simply identify a visual resemblance.
The announcement states that every task was solved by at least two people within a maximum of two attempts in a controlled study. It also reports an average human score of 60%. These findings, published at launch in March 2025, describe different things: collective coverage of the tasks and average individual performance.
That distinction prevents a common misunderstanding. Saying that humans solved every task does not mean that every participant achieved a perfect score.
Human calibration establishes that the problems are accessible under the conditions studied. It does not demonstrate that humans and machines start with exactly the same prior knowledge. A rigorous comparison must examine the instructions, allowed attempts, and participant population, as well as the final average.
ARC-AGI-3: learning through exploration
ARC-AGI-3 presents abstract environments in which the system must act to understand what is happening. Rules and goals are not explicitly provided. An action produces an observation that may support or contradict a hypothesis.
The practical difference is clear: with a static grid, you can revisit the demonstrations already available. In an interactive environment, poorly chosen exploration can consume actions without yielding useful information. The system has to decide what to try and what to remember.
As a teaching example, imagine an object that changes appearance after passing through a region. The system could investigate whether the change depends on the region, the movement, or another element. A useful experiment changes an identifiable factor; a sequence of actions without a hypothesis makes the outcome harder to interpret.
The scoring method presented in the technical report accounts for action efficiency relative to a human reference. In particular, the report applies a quadratic penalty to inefficiency: completing a level with many actions may earn little credit.
It is therefore necessary to distinguish between:
- reaching a successful state;
- reaching it through efficient exploration;
- retaining what was learned for subsequent levels.
This distinction brings the test closer to active learning problems. It does not turn these abstract games into a complete simulation of an agent working in a browser or business application.
Published scores as of October 24, 2026
The official ARC Prize results table, accessed October 24, 2026, lists the following values. This selection illustrates differences between versions; it is not a comprehensive ranking.
| Model | Date displayed by ARC Prize | ARC-AGI-1 | ARC-AGI-2 | ARC-AGI-3 |
|---|---|---|---|---|
| GPT-6.1 Sol | September 29, 2026 | 98.5% | 94.2% | 96.37% |
| Gemini 4 Argon | October 13, 2026 | 98.5% | 91.3% | 97.46% |
| Claude Sonnet 5.5 | September 28, 2026 | 96.5% | 83.9% | 72.44% |
These rows combine results that do not necessarily use the same settings or evaluation setup across all three tests. They provide a starting point for examining the detailed evaluations.
The displayed date belongs to the entry in this table; it is not independent evidence of the model’s commercial launch date. A dash elsewhere in the results indicates that no value has been published. It does not mean zero.
The closely grouped ARC-AGI-1 scores also reveal a practical limitation: this version leaves little room to distinguish among the systems in this selection. Differences remain more visible on the other versions.
Finally, the very low scores reported at the launch of ARC-AGI-3 belong to an earlier stage of evaluation. Presenting them as current performance in October 2026 would give an outdated picture of the published results.
Why the evaluation setup changes the result
An interactive model does not operate on its own. Software passes observations to it, receives its decisions, and retains certain information between actions. This surrounding software is often called a harness.
The official GPT-6.1 Sol evaluation report, accessed October 24, 2026, distinguishes between a Standard setup and a Provider Adapter. The former allows the model to pass along notes it chooses to retain; the latter preserves opaque reasoning state between requests and uses compaction for long conversations.
| Model and setting | ARC-AGI-3 Standard | ARC-AGI-3 Provider Adapter | Caveat |
|---|---|---|---|
| GPT-6.1 Sol, max | 52.7% | 96.2% | Same setting, different state retention |
| GPT-6.1 Sol, xhigh | 39.9% | 96.4% | A different setting also changes the results |
| Gemini 4 Argon, low | 49.8% | 97.5% | Standard evaluation was partially interrupted |
The official Gemini 4 Argon evaluation report explains that API problems prevented some Standard evaluations from finishing. Published scores include unfinished tasks with whatever partial credit they earned. The report also rounds some values shown in the summary table.
The gap does not isolate a single cognitive cause. Above all, it shows that state management is part of the experimental conditions. Comparing a Standard result with a Provider Adapter result without disclosing the difference mixes distinct configurations.
Model, method, and compute: identifying what succeeds
Performance can depend on the model and on how the system uses it. Producing a grid directly, writing a candidate program, and exploring multiple hypotheses are different procedures.
To understand a result, separate three levels:
- The model: its learned parameters and the capabilities available during inference.
- The method: instructions, representations, memory, and answer selection.
- The resources: allowed computation, time, attempts, and tools.
This distinction helps explain improvements. A better method may reveal a capability that was previously used poorly; more compute may allow the system to explore additional possibilities. The score alone does not identify which contribution matters most.
Program synthesis offers one way to represent the work: propose an executable transformation, check that it explains the demonstrations, then apply it to the new input. A solution consistent with the examples can still fail on the unseen case.
Test-time adaptation covers mechanisms that use the problem at hand to improve the response. However, spending more compute on a task does not automatically mean changing the model’s parameters.
For users, the practical question has two parts: what performance does the system achieve, and what resources are actually needed to reproduce it?
What to check before comparing scores
Before comparing results, examine their experimental context. A benchmark becomes informative when its protocol is precise enough to explain what the numbers represent.
Start with the task set. Public sets support inspection and development; semi-private and private sets aim to restrict access to evaluation answers. Success on a public set does not provide the same evidence as success on problems kept confidential.
Next, check the number of answers allowed. Under pass@2, a task can count as solved if either of two final answers is correct. That is not directly comparable to a procedure requiring a single answer.
Benchmark contamination is another limitation. An exact answer may reflect prior exposure rather than fresh adaptation. Even the absence of obvious copying does not rule out every similarity to training data.
Finally, record the harness, tools, budget, and any interruptions. A protocol difference does not invalidate a result, but it limits the comparisons that can be drawn from it.
A useful result description fits into a sentence: a particular system, in a particular configuration, achieved a given score on a specified set, using specified resources, on a particular date. How to read LLM benchmark rankings develops this approach beyond ARC.
What the results can tell us about AGI
Strong ARC-AGI performance supports a bounded claim: the system solves problems in this family under the evaluated conditions. It does not directly measure professional reliability, social understanding, or the ability to conduct independent scientific research.
François Chollet’s perspective places learning efficiency at the center of the definition of intelligence. Other researchers organize evaluation differently, without necessarily disagreeing about every experimental result.
In Levels of AGI for Operationalizing Progress on the Path to AGI, Meredith Ringel Morris and her coauthors propose distinguishing among depth of performance, breadth of capabilities, and autonomy. This framework considers a broader capability profile than a single test can provide.
The approaches do not address exactly the same question. One emphasizes the efficient acquisition of new skills; the other seeks to place capabilities along several dimensions. Presenting an ARC-AGI percentage as a consensus answer to the definition of AGI would overstate what it establishes.
The opposite risk also exists: dismissing every improvement because the test remains incomplete. An instrument can reveal measurable progress without covering the entire phenomenon under study. This balanced reading is what makes the scores useful.
How to use this reference
To get started, examine a task and distinguish the information provided from the rule you need to infer. ARC-AGI explained offers a shorter introduction; this page brings together the distinctions needed to interpret results.
When following research, keep the conditions attached to each score. A leaderboard screenshot quickly loses value if it omits the evaluation set or the harness used.
When choosing a tool for professional use, treat ARC-AGI as a supplementary indicator. A system that performs well on abstract grids still needs to demonstrate that it understands your documents, follows your constraints, and produces verifiable actions.
The competition is also a separate framework. The page on ARC Prize explains its rules and editions; a public model result does not automatically qualify as an eligible entry for its awards.
Key takeaways:
- ARC-AGI examines adaptation to unfamiliar problems within a defined experimental scope.
- Static and interactive versions do not produce directly interchangeable percentages.
- Current results need to be distinguished from historical launch figures.
- The harness, memory, attempts, and compute are part of the result.
- A high score documents a capability; it does not settle the question of AGI on its own.
Related pages
Frequently asked questions
Is ARC-AGI an IQ test for AI?
The comparison helps explain its abstract problems, but ARC-AGI does not produce an IQ score. Results depend on the task set, allowed computation, and evaluation protocol; they cannot be converted into human psychometric scores.
Can I try ARC-AGI without knowing how to code?
Yes. Public interfaces let you solve grids or explore interactive environments in a browser. Programming becomes useful when you want to build a system for automated evaluation, but you do not need it to try the exercises.
Does a high ARC-AGI score prove that an AI is general?
It demonstrates performance within a specific evaluation setting. A claim of general capability also requires evidence of transfer to other domains, an account of the resources used, and robust results on independent tasks.
Why are there different scores for the same model?
Evaluations may use different reasoning settings, task sets, or software that retains information between actions. Check the detailed evaluation report before comparing percentages in isolation.
What is the difference between ARC-AGI and ARC Prize?
ARC-AGI is the benchmark series. ARC Prize refers to the foundation and its associated competitions, whose participation rules and compute limits do not apply to every public model evaluation.