Glossary · AGI and alignment
Generalization
Generalization is an AI system’s ability to use what it has learned to handle new cases. It differs from simply memorizing examples and can involve inputs similar to training data or more unfamiliar situations, where the system’s ability to perform reliably may be less certain.
Generalization: the explanation
During training, a model adjusts its behavior using examples. Generalization occurs when useful learned patterns also work on information not used for that adjustment. Evaluators therefore reserve independent cases and inspect the results. The separation must be meaningful: nearly identical documents or answers already present in training can create a misleading impression of novelty. A good assessment asks whether the system handles new cases, rather than whether it can reproduce something encountered before.
The concept is central to machine learning and directly relevant to everyday tools. An agent trained on certain invoice layouts may need to read a different format. An assistant accustomed to simple requests may encounter combined instructions. The question is which learned patterns remain useful in the new setting. Success on closely related examples does not establish adaptation to every situation. How much the new cases differ from familiar ones matters when interpreting performance.
Distinguish a model’s generalization from the adaptability of the complete application. An agent may succeed by consulting documentation, using software, or asking you to clarify a request. That can be valuable even when the underlying model would fail alone. Conversely, one successful example does not establish a stable capability. A relevant evaluation varies documents, wording, and exceptions, then examines specific failures rather than relying exclusively on an overall score that can hide important weaknesses.
A concrete example
You use an agent to classify invoices. It handles familiar templates, then receives a differently arranged document containing a credit note. Correctly identifying the amounts and document type demonstrates useful generalization. If it asks for clarification instead of guessing when the document is ambiguous, it also manages an important limitation. Both behaviors matter when you decide whether it can support your actual workflow reliably.
Why it matters
Generalization determines whether a tool remains useful as your information changes. Test varied layouts, unusual wording, and representative edge cases before choosing it. Keep some examples separate from the material used to configure the application. Also examine how it communicates difficulty. Strong performance on a prepared demonstration does not guarantee the same results on the changing documents and situations you encounter in practice.
Common misconceptions
- A high score on examples already used to configure a system does not demonstrate generalization, because it may reflect memorized details rather than a reliable ability to handle genuinely new cases outside that preparation material.
- Generalization is not one uniform quality that applies everywhere, because a model may adapt successfully to a layout change while failing on a new language, a different combination of rules, or an unfamiliar operating context.
Related terms: agi and alignment
- AGI (artificial general intelligence)
- AGI, or artificial general intelligence, refers to the idea of AI that can learn and perform a broad range of intellectual tasks beyond a narrow specialty.
- ARC-AGI
- ARC-AGI is a benchmark series designed to study how AI systems solve unfamiliar problems from limited examples or interactions.
- ARC Prize
- ARC Prize is a research and competition initiative that encourages open progress on AI generalization problems.
- fluid intelligence
- Fluid intelligence is the ability to reason through unfamiliar problems without relying mainly on previously learned knowledge.
Where this site covers it
The term appears on 21 pages of this site. A few of them:
- Generalization at the heart of intelligence: François Chollet's framework, examples, and measurement Guide
- How We Get To AGI: François Chollet’s talk, chapter by chapter Guide
- Reasoning models: how they work, what they improve, and measured limits Guide
- Test-time adaptation: how it works and what results show Guide
- The limits of deep learning: what models struggle with, according to François Chollet and other researchers Guide
- Measuring progress toward AGI: benchmarks, limits, saturation, and contamination Guide
- When will AGI arrive? Published forecasts, their methods, and why they differ Guide
- The history of AGI: from Turing to today's labs Guide
Go further
Frequently asked questions
How can you test it simply?
Prepare realistic cases the tool did not encounter during configuration. Vary formats and difficulty without arbitrarily changing the objective. Compare expected outputs, mistakes, and clarification requests across this independent set rather than relying on a single successful demonstration.
Is it the same as learning live?
No. A model can apply previous learning to a new case without changing its parameters. Using examples within a conversation is adaptation through context. It does not necessarily mean the system has acquired a lasting new skill.
Can an agent outperform a model alone?
The full system may succeed more often through tools, document retrieval, and verification. Be clear about what is being evaluated. This practical improvement does not automatically demonstrate a change in the intrinsic capabilities of the underlying model.
Neutrality. No affiliate links. No vendor pays to be reviewed, rated or ranked. Reviews are directed by François Chollet, neutral and factual, and every data point is dated and sourced. How we score