Interpretability: opening the black box of AI models
Interpretability is the research field that tries to understand how an AI model really reaches its answer, by examining its internal computations rather than what it says about itself. It is an expert topic, but it shapes how far we can trust the most capable systems.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
What it means
In one sentence: interpretability is the attempt to read a model’s mind. A large language model is a neural network made of billions of numbers tuned during training; nobody wrote them by hand, and nobody knows up front what they encode. Mechanistic interpretability looks inside a Transformer’s activations for concepts (features) and for the chains of computation that connect them (circuits).
The core obstacle is that a single neuron rarely stands for a single idea. It fires for several unrelated concepts, because the model packs more concepts than it has neurons, a phenomenon called superposition. Researchers therefore use dictionary learning, typically sparse autoencoders, to recover more readable units. In the “Towards Monosemanticity” study published by Anthropic in October 2023, this approach surfaced features in a small model dedicated to things like DNA sequences, legal language and HTTP requests.
A concrete example
In work published in March 2025 on tracing the thoughts of Claude, Anthropic describes attribution graphs that follow the path of a computation. Two striking findings: when writing a rhyming line, the model picks the target rhyme before composing the line; when adding numbers, it runs a rough estimate and an exact last-digit calculation in parallel, yet when asked, it describes the carry-the-one method taught at school.
Why it matters
- Safety and alignment: checking that a model is not pursuing a hidden goal means inspecting its mechanisms, not just its outputs.
- Debugging: understanding why a model produces a hallucination or falls for manipulation helps fix the root cause.
- A complement to testing: red teaming probes behavior from the outside; interpretability tries to explain that behavior from the inside.
Common misconceptions
- “The displayed reasoning is what the model thinks.” Not guaranteed: the written explanation can diverge from the actual computation.
- “We can already read everything.” Analyses cover fragments of behavior, at high cost, and do not yet scale to an entire model.
- “It’s purely academic.” These tools already shed light on very practical behaviors: why a model refuses, why it makes up an answer, or when it constructs reasoning to justify a conclusion it was nudged toward.
Related terms: alignment, red teaming, AI bias, constitutional AI.
Related terms
- AI alignment · The research and methods aimed at making an AI system actually pursue the intentions and values of the people who build and use it.
- Neural network · Software made of many small connected computing units that learns by adjusting the strength of the links between them.
- Transformer · A neural network architecture built on attention, introduced in 2017, that underpins almost every large language model today.
- Red teaming · Adversarial testing in which people or other AIs deliberately attack a model to uncover its weaknesses before real attackers do.
- AI bias · A systematic error that makes an AI treat some groups unfairly or repeat stereotypes, often inherited from its training data.
- Constitutional AI · An Anthropic training method in which an AI critiques and revises its own answers against a written list of principles called a constitution.
Full glossary
Frequently asked questions
Is interpretability the same as explainability?
The words are often used interchangeably. In practice, explainability usually means explanations given to users about a decision, while interpretability, especially mechanistic interpretability, studies the network's internal mechanisms directly.
Can't we just ask the model to explain its reasoning?
Not reliably. Research published by Anthropic in March 2025 shows cases where the model describes a different calculation method from the one it actually uses internally. The written explanation is generated text like any other, not a readout of the computation.
What is a feature in interpretability research?
A feature is a direction in the model's activation space that corresponds to an identifiable concept, such as code, a language or a legal notion. Researchers extract features with dictionary learning methods such as sparse autoencoders.
Is interpretability a solved problem?
No. Current methods illuminate part of the mechanisms, on specific tasks, and require substantial analysis effort. Understanding everything a large model does remains an open research goal.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.