Glossary · AGI and alignment
ARC-AGI
ARC-AGI is a benchmark series designed to study how AI systems solve unfamiliar problems from limited examples or interactions. Its versions use different tasks and evaluation rules, and their results should not be treated as a certification that a system has achieved artificial general intelligence.
ARC-AGI: the explanation
In the static grid versions, a system sees examples pairing an input grid with a transformed output grid. It must infer the rule and produce the output for a new input. Rules may involve shapes, positions, or colors. The challenge is selecting a transformation consistent with the examples. This format reduces reliance on encyclopedic knowledge, but it does not remove the influence of training, learned representations, or the methods used to search for solutions.
The series investigates adaptation to unfamiliar tasks. Its formats have evolved: ARC-AGI-3 includes interactive environments in which actions and their effects help reveal the problem. You therefore need the version before comparing scores. The official overview distinguishes these formats and explains the research emphasis on generalization: [ARC-AGI overview](https://arcprize.org/arc-agi). A shared series name does not make results interchangeable. Different interaction patterns can test different aspects of learning and problem solving under the evaluation rules.
A score needs its conditions: evaluated tasks, possible exposure to examples, permitted attempts, and resources used. A solution may combine a model, generated code, and a search procedure. In that case, the result belongs to the complete system. Even remarkable success remains bounded by the test’s scope. It does not directly establish that the system can manage your inbox, understand every human situation, or use connected tools reliably and safely in everyday work.
A concrete example
You see a strong result advertised on ARC-AGI. Before concluding that the agent suits your work, you look for the benchmark version and evaluation method. A system may solve grid transformations effectively while struggling with your invoices. You therefore test representative documents and constraints as well. The benchmark provides evidence about certain abilities, which you supplement with an assessment of the work you actually need.
Why it matters
ARC-AGI contributes to discussion of generalization beyond memorized answers. Its value depends on careful comparison, however. Check the version, task set, and solving budget before interpreting a ranking. When selecting a tool, combine that information with practical trials. A higher score does not by itself guarantee stronger confidentiality, fewer document errors, or better reliability in the activities you want the agent to perform.
Common misconceptions
- Success on ARC-AGI is not automatically proof of AGI, because the benchmark measures particular kinds of adaptation within a defined setting and does not cover every ability, constraint, or situation associated with general intelligence.
- Two results labeled ARC-AGI are not necessarily comparable, because benchmark version, task selection, allowed attempts, and computing resources may differ and substantially change what the reported scores mean for someone assessing the systems.
Related terms: agi and alignment
- AGI (artificial general intelligence)
- AGI, or artificial general intelligence, refers to the idea of AI that can learn and perform a broad range of intellectual tasks beyond a narrow specialty.
- ARC Prize
- ARC Prize is a research and competition initiative that encourages open progress on AI generalization problems.
- generalization
- Generalization is an AI system’s ability to use what it has learned to handle new cases.
- fluid intelligence
- Fluid intelligence is the ability to reason through unfamiliar problems without relying mainly on previously learned knowledge.
Where this site covers it
The term appears on 94 pages of this site. A few of them:
- ARC-AGI: the test, its versions, and published scores Guide
- How We Get To AGI: François Chollet’s talk, chapter by chapter Guide
- ARC Prize: the competition, its rules, editions, results, and lessons Guide
- Generalization at the heart of intelligence: François Chollet's framework, examples, and measurement Guide
- Measuring progress toward AGI: benchmarks, limits, saturation, and contamination Guide
- The limits of deep learning: what models struggle with, according to François Chollet and other researchers Guide
- Program synthesis: principles, history, and a possible path to AGI Guide
- Test-time adaptation: how it works and what results show Guide
Go further
Frequently asked questions
Is it a knowledge quiz?
It primarily targets rule inference and adaptation rather than recall of encyclopedic facts. That does not make prior learning irrelevant. Acquired representations and solution methods still influence performance, so the evaluation setup remains important when interpreting a result.
Why specify the version?
Versions may use different tasks and interaction rules. A static grid result should not be read as equivalent to a result in an interactive environment. The version is an essential part of reporting the measurement accurately.
Does a score predict your workflow?
It offers evidence about the abilities targeted by the benchmark, rather than a complete prediction. For your workflow, also examine representative tasks, failure patterns, response times, and permissions. Those practical factors may lead to a different choice.
Neutrality. No affiliate links. No vendor pays to be reviewed, rated or ranked. Reviews are directed by François Chollet, neutral and factual, and every data point is dated and sourced. How we score