Knowledge distillation: when a big model teaches a small one
Knowledge distillation means training a small “student” model to reproduce the behaviour of a large “teacher” model. The goal is a much lighter, cheaper model that keeps a good share of the original's abilities.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|---|
| Origin | 2015 |
| Introduced by | Hinton, Vinyals et Dean |
Teacher and student
A large model knows a lot, but it is expensive to run. Distillation starts from a simple idea: instead of training a small model only on raw data, you have it imitate a large model that already performs well. The student learns not just the right answer but also how the teacher spreads its confidence across possible answers, which carries a surprising amount of information.
That is the idea set out by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in a 2015 paper: squeeze the knowledge of an ensemble of models into a single model that is simpler to deploy.
A concrete example
In 2019, Victor Sanh and co-authors released DistilBERT, a distilled version of the BERT model. According to their paper, it was 40% smaller and 60% faster while keeping 97% of the original’s language-understanding capabilities.
Today the technique is everywhere in large language models. A lab uses its biggest model to generate a large volume of answers or reasoning traces, a form of synthetic data, then trains smaller models on them. Many small language models and open-weights models are described as “distilled” from a larger one. You can browse some of them on our open-source models page.
Why it matters
- Cost: a model with fewer parameters answers faster and costs less per request.
- Running locally: a distilled model can run on a laptop or phone, without sending your data to the cloud.
- Specialisation: you can distill a single skill (summarising, classifying, extracting) into a compact model.
Common mistakes and myths
- “Distilled means compressed with no loss.” There is almost always some loss, especially on complex or unusual tasks.
- “Distillation and quantization are the same thing.” Quantization lowers the numerical precision of the same model; distillation trains a new, smaller model.
- “The student inherits everything, reliability included.” It also inherits the teacher’s errors and biases, and may add its own.
- “It is always allowed.” Some providers’ terms of use may forbid it: it is a licensing question, not just a technical one.
Related terms: fine-tuning, small language model, synthetic data, quantization.
Related terms
- Small language model (SLM) · A compact language model, light enough to run on a laptop or phone, that performs well on focused tasks rather than everything at once.
- Fine-tuning · Continuing the training of an already trained model on a small, targeted set of examples to adapt it to a task, style or output format.
- Synthetic data · Artificially created data, often generated by another AI model, used to train, fine-tune or test a model when real data is scarce.
- Quantization · A technique that stores a model's weights with fewer bits (8, 4 or even fewer) so it needs far less memory to run.
- Model parameters · The internal numbers a model tunes during training; how many it has indicates its size, not directly how good it is.
- Open-weights model · An AI model whose trained parameters (the weights) can be downloaded, so you can run it on your own hardware.
Try it
Full glossary
Frequently asked questions
How is distillation different from fine-tuning?
Fine-tuning continues training a model on chosen data, often written by people. Distillation uses another, larger model as the source of examples or probabilities to imitate. In practice, distillation often involves fine-tuning the student on the teacher's outputs.
Is a distilled model as good as its teacher?
Usually not, but the gap can be small on the target tasks. DistilBERT, for instance, was reported by its authors to be 40% smaller and 60% faster than BERT while retaining 97% of its language-understanding capabilities. On harder tasks the difference shows more.
Can I distill any model I like?
Technically, you only need to be able to query the teacher. But the terms of use of some commercial services may forbid using their outputs to train a competing model. Check the teacher model's licence and terms before you start.
Who introduced knowledge distillation?
The term took hold with “Distilling the Knowledge in a Neural Network” by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, posted to arXiv in March 2015. They showed how to compress the knowledge of an ensemble of models into a single model that is easier to deploy.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.