On the Measure of Intelligence, explained

In On the Measure of Intelligence, François Chollet proposes measuring AI intelligence by how efficiently a system acquires new skills, accounting for its prior knowledge, experience, and the difficulty of the tasks. Measuring AI intelligence therefore requires more than a success rate: it means examining how the system becomes competent when faced with an unfamiliar problem, as set out in the paper submitted to arXiv on November 5, 2019.

Task success does not tell the whole learning story

François Chollet’s central argument distinguishes the skills a system has acquired from its ability to acquire skills. Excellent performance may depend on specialized preparation, knowledge built in by the designers, or a large number of examples. His framework accounts for the scope of the tasks, prior knowledge, experience, and the difficulty of generalization.

Consider a hypothetical example. A program correctly extracts information from a form where every field has a known position. You change the layout, move the sections, and introduce a new convention explained through a few examples. Success on the original form alone does not tell you how the program will respond.

This example does not diminish the value of specialized automation. If you only need to process the original form, reliability remains the decisive criterion. If you expect the system to adapt to unexpected documents, however, you need to test that capability separately.

A useful measurement therefore starts with a precise question: what property are you trying to observe? Immediate success, resilience to layout changes, and learning a new rule require different tests. Combining them makes the result difficult to interpret, even when the score itself is calculated correctly.

What you need to know to interpret a result

The paper’s terminology becomes easier to follow when expressed as evaluation questions. The following table is a guide to understanding the framework, rather than a formula for calculating a universal intelligence score.

Factor to examine	Question to ask	Example in a document-processing test
Scope	Which families of problems does the result cover?	Forms, tables, letters, or a combination of these documents
Prior knowledge	What information is available before the test?	Document templates, business rules, instructions
Experience	What new information is provided during learning?	Corrected demonstrations, attempts, and feedback
Generalization difficulty	What must the system discover beyond the examples?	A new convention or a combination of rules

Suppose you introduce an unfamiliar classification rule to a system. You provide documents that have already been classified, followed by documents for it to process. If the instructions explicitly spell out the entire procedure, you are mainly evaluating its ability to follow that procedure. If the examples leave a pattern for it to discover, you are observing something different.

The distinction depends on the content of the test. A previously unseen document may follow a structure the system has already mastered. Conversely, a visually familiar document may require an unfamiliar operation.

This helps explain the significance of generalization. When designing your test, describe what changes between preparation and evaluation: the data, the presentation, the rule, or the objective. Simply calling a task new is too vague to define a testing protocol.

Learning efficiently means more than responding quickly

The official ARC Prize overview uses the concept of skill-acquisition efficiency and distinguishes prior knowledge and experience as resources. It draws on fluid intelligence: solving unfamiliar problems and adapting, rather than only recalling accumulated knowledge.

When reading a result, avoid reducing efficiency to execution speed. A system may respond quickly after extensive preparation. Another may take longer during the test while using fewer examples specific to the task.

Consider a hypothetical experiment in which you teach a sorting convention using objects with different shapes. You observe the responses after each demonstration, then introduce a combination that has not appeared before. You can record how much help the system needs, which errors persist, and whether it can reuse the convention.

Elapsed time remains useful information, but it cannot replace these observations. In a professional setting, you would also include cost and computing resources consumed. Those dimensions help assess the system’s practical usefulness; combining them into an opaque score makes that assessment harder.

A clear report therefore keeps several measures side by side. This lets you distinguish improved accuracy, a reduced need for demonstrations, and faster processing. Our page on generalization and how to evaluate it explores this approach further.

ARC makes the question observable through grids

François Chollet accompanies his argument with ARC, the Abstraction and Reasoning Corpus, a collection of problems. The official overview of the original corpus, now called ARC-AGI-1, describes its purpose: testing the ability to solve problems for which a system has not been specifically prepared.

The official repository describes tasks consisting of input and output grid pairs that serve as demonstrations. The participant must construct the output for a new input, including determining its dimensions. A correct answer must match the expected grid exactly.

Here is an invented illustration of the mechanism, rather than an official task: the demonstrations show a shape moving until it touches a marker. In the new input, the shape’s position and size change. Copying a previous output fails; the participant must identify the relevant transformation.

The educational value lies in making the difference between recognizing an appearance and applying a relationship visible. Several explanations may fit a single example. Additional demonstrations should help distinguish between those hypotheses.

The format still has a clear boundary: solving the transformation establishes success on that problem. It does not directly demonstrate an ability to negotiate, manage a project, or operate software. ARC-AGI explained describes how the test works without treating its score as an overall verdict on a system.

Prior knowledge does not disappear

ARC Prize explains that its tasks are designed around basic knowledge, such as objects, space, and quantities, rather than specialized cultural knowledge. This approach aims to support a meaningful comparison with human adaptability. It does not assume participants begin with no knowledge at all.

One methodological consequence deserves explicit attention: the starting point matters as much as the visible instructions. In an experiment, you need to distinguish the knowledge participants are allowed to bring, the knowledge you intend to teach, and the knowledge that would make the task trivial.

Imagine a test that requires discovering how to group objects. If you provide a tool that already performs exactly that grouping, success takes on a different meaning. You can still measure the ability to use the tool, but you have shifted where the difficulty lies.

The same issue arises with highly detailed instructions. A procedure written by the evaluator may solve most of the problem before the first attempt. To understand the system’s contribution, retain the instructions and examples in the evaluation report.

This transparency also makes results easier to compare over time. If you change the model, tools, and instructions simultaneously, a higher score describes the resulting system as a whole. It does not automatically establish that the model alone improved. The appropriate unit of evaluation depends on your question: an individual component or the complete setup the user will work with.

Design a test that resists shortcuts

The ARC repository asks developers to prevent evaluation tasks from influencing algorithm development. In particular, it identifies the risk of repeatedly modifying a system using its evaluation score as feedback. This precaution protects the separation between preparation and measurement.

The official ARC Prize testing policy also places trust in the process, reproducibility, and the ability to audit methods at the center of score interpretation.

For your own evaluation, you can turn these principles into a straightforward procedure:

• Define the capability you want to assess before creating examples.
• Set aside tasks that will not be used to refine the instructions.
• Specify the assistance allowed and the stopping conditions.
• Keep a record of outputs, errors, and human interventions.
• Publish the testing conditions alongside the result.

Add variations that reflect your needs. If you are assessing adaptation to a new rule, changing only a document’s color may not be enough. If you are assessing whether a system can read documents reliably despite changes in appearance, that variation may be relevant.

Also decide how to handle partially correct answers. An overall ranking may hide an error in a rare but consequential category. Reporting results by task family makes that weakness visible and supports better decisions.

The guide to evaluating an AI agent extends this approach to systems that take actions. For help interpreting public score tables, see how to read model leaderboards.

What this approach helps you decide

The distinction between skill and learning primarily changes how far your conclusions can reach. A specialized test may be sufficient to select a solution for a stable task. A claim about general intelligence requires examining a variety of situations and how the system acquires its skills.

For a business purchase, start with your actual requirements. You may need reliable processing of recurring documents, with identifiable errors and an acceptable cost. Requiring a demonstration of general intelligence would add little to that limited decision.

If your project depends on frequent changes, include adaptation scenarios. Introduce an unfamiliar convention, observe how much assistance is needed, and check whether the system transfers what it has learned to cases held back for evaluation. Define in advance what would count as a usable result.

Keep the limits clear, however: your local experiment does not become a universal measure of intelligence by extension. It provides a bounded answer, often more useful for a decision than a broad label. The AGI resources place these questions within the wider research debate.

The main practical contribution of this framework is a discipline for reasoning: specify the property being measured, document the conditions, and limit the conclusion to what the test establishes. You can recognize a useful skill while leaving open the question of how broadly it generalizes.

Key takeaways

• Performance describes what a system achieves under the conditions of a test.
• François Chollet proposes examining skill-acquisition efficiency, accounting for the prior knowledge and experience required.
• ARC makes this question observable within a defined setting, without providing a universal certificate of intelligence.
• To evaluate an AI system, document the assistance provided, hold back unfamiliar tasks, and keep your conclusions proportional to the results.

Frequently asked questions

How does François Chollet define intelligence?

François Chollet proposes evaluating intelligence through the efficiency with which a system acquires new skills. That efficiency must be assessed across a range of tasks, accounting for prior knowledge, experience, and the difficulty of generalization.

Does a high benchmark score prove that an AI is intelligent?

It establishes performance under the conditions of that test. Interpreting it as evidence of general intelligence also requires examining how unfamiliar the tasks were and how much preparation the system received.

Is ARC-AGI an IQ test for AI?

The comparison helps explain its focus on unfamiliar problems, but treating the tests as equivalent would be misleading. An ARC-AGI result cannot be converted directly into a human IQ score.

Should you choose AI tools based on their ARC-AGI scores?

The score can shed light on a particular capability, but choosing a tool requires an evaluation suited to your work. Also check its errors, the human intervention it needs, turnaround times, and the cost of successfully completed tasks.