Speculative decoding: making an LLM write faster without changing its answers
Speculative decoding speeds up a large language model: a small, fast model guesses the next few words and the large model checks all of those guesses at once instead of writing them one by one. Done properly, it makes generation faster without changing the distribution of the answers.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|---|
| Origin | 2022 |
| Introduced by | Yaniv Leviathan, Matan Kalman, Yossi Matias |
The idea in one sentence, then the details
Speculative decoding lets a fast helper write a rough draft that the main model reviews and approves in one go.
A language model normally produces one token at a time, and every token requires a full pass through the large model. That work is often limited by how fast memory can be read rather than by raw compute: the GPU spends more time waiting than calculating. Speculative decoding uses that slack:
- a small, fast draft model proposes several tokens ahead;
- the target model scores all of those proposals in a single parallel pass;
- it accepts the longest prefix consistent with what it would have produced, fixes the first rejected token, and the cycle starts again.
A rejection-sampling rule guarantees that the output follows exactly the large model’s distribution. The idea was published in November 2022 by Yaniv Leviathan, Matan Kalman and Yossi Matias in the original arXiv paper; Chen and co-authors described a closely related method, speculative sampling, in February 2023.
A worked example
The model has to complete “The capital of Italy is”. The draft model proposes “Rome , a city”. The large model checks all four tokens in one pass: it agrees with “Rome” and the comma, but would have written “located” rather than “a”. It keeps the first two, replaces the third, and a new cycle begins. Three tokens for a single pass of the large model instead of three.
Why it matters
Latency is critical for agents and voice assistants, which chain many calls together. Leviathan and co-authors report a 2 to 3 times speed-up over the standard T5X implementation with identical outputs; Chen and co-authors report 2 to 2.5 times on Chinchilla 70B. The real gain depends on the acceptance rate: on predictable text such as code or a repetitive format, the draft guesses right more often. The technique is now built into inference servers such as vLLM and llama.cpp.
Common myths
- “The small model answers instead of the big one.” No: it only proposes, and the large model always has the final say.
- “It is always faster.” If the draft is often wrong, or the server is already saturated with requests, the gain shrinks or even reverses.
- “It reduces total compute.” Quite the opposite: a little more is computed. You save waiting time, not operations.
Related terms: inference, KV cache and small language model.
Related terms
- Inference · The moment a trained model is actually used to produce an output: every question you send to an AI triggers an inference run.
- Latency · The waiting time between sending a request to an AI model and receiving its answer, from the first word through to the last.
- Token · A small chunk of text (a word, part of a word or a symbol) that an AI model reads and writes; the unit for limits and pricing.
- KV cache · Memory where a model keeps the work already done on previous tokens so it does not have to redo it for every new word.
- Inference server · Software that loads an AI model into memory and answers requests through an API, while optimizing throughput and GPU memory.
- Small language model (SLM) · A compact language model, light enough to run on a laptop or phone, that performs well on focused tasks rather than everything at once.
Full glossary
Frequently asked questions
Does speculative decoding hurt quality?
Not in its original form. The 2022 and 2023 papers use an acceptance rule that guarantees the output follows the same distribution as the large model alone. Some so-called approximate variants relax that guarantee to go faster.
Do I need a second model?
Often, yes: a small model from the same family that shares the same tokenizer. Some variants do without one, for example by looking up word sequences already present in the prompt (n-grams) or by adding small prediction heads to the main model, as EAGLE does.
Which software supports it?
As of October 22, 2026, both the vLLM documentation and the llama.cpp server documentation list speculative decoding among their features.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.