Mixture of experts (MoE): a huge model that only works in part
A mixture-of-experts (MoE) model holds many specialised sub-networks but only calls on a few of them for each token. That lets it be very large on paper while staying faster and cheaper to run than a conventional model of the same size.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|---|
| Origin | 2017 |
| Introduced by | Shazeer et al. |
The idea: many experts, few on duty
In a conventional “dense” model, every parameter takes part in computing every token. In a mixture-of-experts model, some layers are replaced by a set of smaller networks, the experts, sitting behind a router (also called a gating network). For each token, the router picks the few experts that fit best, and only those do any work. The rest stay idle.
The approach was popularised for language by Shazeer and co-authors in a 2017 arXiv paper describing “conditional computation”: the network’s capacity grows enormously while the compute per example grows far less.
A concrete example: Mixtral 8x7B
In December 2023, Mistral AI released Mixtral 8x7B, an open-weights model under the Apache 2.0 licence. According to the official announcement, each layer holds 8 groups of parameters (the experts) and the router selects 2 of them per token. The result: 46.7 billion parameters in total, but only 12.9 billion used for each token. In practice the model runs at roughly the pace of a 13-billion-parameter model while having far more capacity to draw on. You can compare models like this on our open-source models page.
Why it matters
- Cost and speed: at similar quality, an MoE needs less compute per token during inference, which cuts both the bill and the wait.
- Scaling up: it lets labs grow the total parameter count without training compute rising at the same pace.
- Reading model specs: many recent models, especially open-weights ones, use this architecture. Knowing the difference between total and active parameters keeps you from making unfair comparisons.
Common mistakes and myths
- “A 46-billion MoE behaves like a 46-billion dense model.” Not necessarily. Its quality often lands somewhere between a dense model the size of its active parameters and one the size of its total parameters.
- “It fits in less memory.” No: all experts must be loaded. That matters a lot if you want to run it on your own hardware.
- “Each expert masters one subject.” The split learned by the router is not thematic in any human sense.
- “MoE is an alternative to the Transformer.” Quite the opposite: in large language models, experts usually replace certain layers inside a Transformer.
Related terms
Model parameters, inference, Transformer, scaling laws and open-weights models.
Related terms
- Model parameters · The internal numbers a model tunes during training; how many it has indicates its size, not directly how good it is.
- Inference · The moment a trained model is actually used to produce an output: every question you send to an AI triggers an inference run.
- Transformer · A neural network architecture built on attention, introduced in 2017, that underpins almost every large language model today.
- Open-weights model · An AI model whose trained parameters (the weights) can be downloaded, so you can run it on your own hardware.
- Training compute (FLOP) · The total number of arithmetic operations, measured in FLOP, needed to train an AI model.
- Scaling laws · Measured relationships that predict how a model's performance improves with its size, its training data and its compute.
Try it
Full glossary
Frequently asked questions
What do total parameters and active parameters mean?
Total parameters are every weight in the model, all experts included. Active parameters are the ones actually used to compute a given token. For Mixtral 8x7B, Mistral AI reported 46.7 billion parameters in total, of which 12.9 billion are used per token.
Does an MoE model need less memory?
Not proportionally. Every expert must stay loaded, because the router may pick any of them for the next token. The saving is mainly in compute per token, which means faster and cheaper inference.
Is each expert specialised in a topic, like law or cooking?
Not really. The router learns on its own how to distribute tokens, and the resulting specialisation is usually hard for humans to interpret. Do not picture a panel of subject-matter specialists.
Where does MoE for language models come from?
The landmark paper is “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, posted to arXiv in January 2017 by Noam Shazeer and co-authors. It showed that a layer of selectively activated experts could massively increase a network's capacity without a matching increase in compute.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.