KV cache: the working memory of a language model
The KV cache is a language model's working memory: it keeps the calculations already done on earlier text so they are not repeated for every new word. It makes generation much faster, but it takes up space that grows with the length of the conversation.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
The definition
The KV cache saves a model from rereading and recomputing all the previous text every time it writes a new word.
A model generates its answer one token at a time. At each step, the attention mechanism compares the current token with every earlier one using two vectors computed for each of them: a key and a value. Because a token never depends on the ones that come after it, those keys and values never change once computed. So they are kept in memory, layer by layer: that is the key-value cache. At the next step, the model only computes the key and value for the new token and appends them.
A worked example
Hugging Face’s documentation walks through a 15.5-billion-parameter model. For a hypothetical sequence of 16,000 tokens, its KV cache would hold close to 8 billion numbers, roughly 15 GB in 16-bit precision, or about half the size of the model’s weights. That same model uses multi-query attention, which would shrink the cache to under 400 MB. This is why a model’s advertised context window does not tell the whole story: the memory actually available on the machine matters just as much.
Why it matters
The KV cache sits at the heart of inference economics. Several innovations aim to shrink it:
- multi-query attention (MQA), proposed by Noam Shazeer in 2019, shares a single key-value pair across all attention heads;
- grouped-query attention (GQA), published by Ainslie and co-authors in 2023, strikes a middle ground with a few shared pairs;
- PagedAttention, introduced in 2023 with vLLM, splits the cache into pages like an operating system’s virtual memory; its authors report 2 to 4 times higher throughput at the same latency.
The same cache is what prompt caching reuses across requests, which is why many APIs charge less for repeated input tokens.
Common myths
- “The cache changes the answers.” In principle it gives the same result as a full recomputation; Hugging Face only notes tiny possible differences due to numerical kernels.
- “Memory is all about the weights.” With long contexts and many concurrent users, the cache can outgrow the weights.
- “It is the assistant’s memory.” No: the KV cache lives for one generation or technical session. An assistant’s memory feature is something else entirely.
Related terms: long context, VRAM and inference server.
Related terms
- Attention mechanism · The technique that lets a model weigh, for each word it processes, how relevant every other word in the text is.
- Context window · The maximum amount of text (in tokens) a model can read and produce at once, including its own answer.
- Long context · A model's ability to process very long inputs in one go, such as books or code repositories, and the techniques that make it possible.
- Prompt caching · Reusing the start of a prompt the model has already processed, so repeated requests run faster and the reused part is billed at a lower rate.
- VRAM (video memory) · The memory built into a graphics card, where an AI model must fit to run fast; it is often what limits what you can run locally.
- Inference server · Software that loads an AI model into memory and answers requests through an API, while optimizing throughput and GPU memory.
Try it
Full glossary
Frequently asked questions
Is the KV cache the same as prompt caching?
They are related but different. The KV cache exists during every generation. Prompt caching, offered by some APIs, keeps that cache between two requests that start with the same text, which is why repeated input tokens can be billed at a discount.
Why does the KV cache limit context length in practice?
Its size grows linearly with the number of tokens and with the number of requests served in parallel. On a server, it is often the cache, more than the weights, that determines how many users or long documents fit in GPU memory.
Can its size be reduced?
Yes: through model architecture (multi-query or grouped-query attention), by quantizing the cache, or with smarter memory management such as PagedAttention in vLLM.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.