Small language models (SLMs): AI that fits on your own device
A small language model (SLM) is a text model far more compact than the large models behind online assistants. It can often run directly on a laptop, phone or modest server, at the cost of less knowledge and weaker complex reasoning.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|
Small, but what for?
The large language models behind consumer assistants run in vast data centres. A small language model takes the opposite bet: fewer parameters, so less memory and compute, in order to run wherever you are, on a laptop, a phone or a small company server.
There is no official definition in terms of parameter count. The term is relative: what counts as “small” today would have been considered large only a few years ago.
A concrete example: the Phi-3 family
In April 2024, Microsoft introduced Phi-3, a family of small models. The most compact, Phi-3-mini, has 3.8 billion parameters. Microsoft highlighted specific uses: running on the device with no connection, keeping data on-premises, answering quickly without a round trip to the cloud, and serving areas with poor internet access.
Picture a medical practice that wants to summarise consultation notes automatically. An SLM installed on an internal workstation avoids sending that sensitive text to an outside service. You can find compact open-weights models on our open-source models page.
Why it matters
- Cost: less compute per answer means a lower bill or more modest hardware.
- Privacy: run locally, the model does not send your data anywhere.
- Speed: no queue and no network trip, which cuts latency.
- Specialisation: a small model tuned for one precise task (sorting emails, extracting fields) can be more than enough.
Small models are often produced by distillation from a larger model, then slimmed down further with quantization so they fit in less memory.
Common mistakes and myths
- “Small means bad.” On focused tasks, a well-chosen SLM can be perfectly adequate. Its limits show on long reasoning and specialist knowledge.
- “It runs on any device.” Even a small model needs enough memory; check the recommended specs.
- “Local means risk-free.” It can still be wrong or make things up: hallucinations do not vanish with size.
- “You have to choose between small and large.” Many systems combine both, with the small model handling simple requests and the large one taking the hard cases.
Related terms
- Large language model (LLM) · A program trained on huge amounts of text to predict what comes next in a text; it is the engine behind AI assistants.
- Knowledge distillation · A technique that trains a small student model to imitate the outputs of a large teacher model, so it becomes lighter and cheaper to run.
- Quantization · A technique that stores a model's weights with fewer bits (8, 4 or even fewer) so it needs far less memory to run.
- Model parameters · The internal numbers a model tunes during training; how many it has indicates its size, not directly how good it is.
- Open-weights model · An AI model whose trained parameters (the weights) can be downloaded, so you can run it on your own hardware.
- Fine-tuning · Continuing the training of an already trained model on a small, targeted set of examples to adapt it to a task, style or output format.
Try it
Full glossary
Frequently asked questions
How small does a model have to be to count as an SLM?
There is no official threshold. The term describes models clearly more compact than the biggest of their day, so the line keeps moving. In April 2024, for example, Microsoft presented Phi-3-mini, with 3.8 billion parameters, as a small language model.
Can I use an SLM without an internet connection?
Yes, that is one of the main attractions. Once the model is downloaded, local runtime software lets you use it offline on a machine with enough memory. Your text then never leaves your computer.
Can an SLM replace ChatGPT or Claude?
Not for everything. Microsoft itself notes that these models suit simpler, focused tasks but are less suited to in-depth knowledge retrieval or complex multi-step reasoning. Many teams mix small and large models depending on the job.
What makes a good small model?
The levers most often cited are carefully curated training data, distillation from a larger model and training focused on the target tasks. Quantization can then shrink the memory needed to run it even further.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.