Latency: why AI sometimes takes its time to answer
Latency is the time between sending a request to an AI model and getting its answer back. In practice there are two parts that feel very different: the delay before the first word appears and the total time to finish the response.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|
One definition, two experiences
Anthropic’s documentation defines latency as the time the model takes to process a prompt and generate an output. It depends on the size of the model, the complexity of the request and the infrastructure running it. Two measurements come up again and again:
- Time to first token (TTFT): how long before the answer starts to appear.
- Total generation time: how long it takes to produce the whole answer, which grows with its length, because a language model writes one token after another.
A concrete example
Picture a customer-service voice assistant. If the first sentence arrives quickly, the conversation feels natural even when the full answer takes longer. A long silence before the first word, on the other hand, makes the system feel broken, however good the final answer is.
For batch work, such as summarising thousands of documents overnight, the latency of each answer matters far less: total cost and throughput come first. To shortlist models by price and capabilities before timing them on your own requests, use our model comparison tool.
Why it matters
Latency is usually a trade-off against quality and cost:
- a large model or a reasoning model often gives better answers, but more slowly;
- a very long prompt adds reading time during inference;
- streaming does not speed up the computation, but it makes the app feel far more responsive.
On the technical side, prompt caching avoids reprocessing the same long context, and providers use techniques such as speculative decoding to generate faster.
Common mistakes and myths
- “The fastest model is the best choice.” Only if its quality is good enough for the task. Anthropic’s documentation actually recommends getting a prompt that works well first and only then optimising for latency.
- “Asking for a short word count is enough.” Models count in tokens; limiting the number of sentences or paragraphs usually works better.
- “A model’s latency is fixed.” It varies with text length, time of day, server load and settings such as the reasoning effort level.
- “Streaming makes the model faster.” It makes the wait more pleasant without reducing total compute time.
Related terms
- Inference · The moment a trained model is actually used to produce an output: every question you send to an AI triggers an inference run.
- Token · A small chunk of text (a word, part of a word or a symbol) that an AI model reads and writes; the unit for limits and pricing.
- Reasoning model · A language model trained to think step by step before answering: slower and pricier, but stronger at maths, code and logic.
- Test-time compute · The computing power a model spends while answering, for instance to think for longer or to compare several candidate solutions.
- Prompt caching · Reusing the start of a prompt the model has already processed, so repeated requests run faster and the reused part is billed at a lower rate.
- Speculative decoding · A speed-up technique where a small model drafts several tokens ahead and the large model checks them all in a single pass.
Try it
Full glossary
Frequently asked questions
What is time to first token (TTFT)?
It is the delay between sending the request and receiving the first token of the answer. With streaming, it is what users feel most: once text starts appearing, the wait seems shorter. Anthropic's documentation treats it separately from the model's baseline latency.
Why are reasoning models slower?
Because they first produce internal reasoning, sometimes lengthy, before the visible answer. Those extra tokens take time to generate. That is the trade-off of test-time compute: more thinking, often better answers, but a longer wait.
How can I reduce latency in my app?
The usual levers are choosing a faster model, shortening both the prompt and the expected answer, and turning on streaming so text shows up as it is written. Prompt caching can also speed up requests that repeat a long context.
Does latency depend on my internet connection?
Only partly. The network trip adds a little delay, but most of it usually comes from the model's compute time, the amount of text to read and write, and how busy the provider's servers are at that moment.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.