Inference: the moment AI answers you
In AI, inference means using an already trained model to produce an output: an answer, an image, a translation. Every message you send to an assistant triggers an inference run, and that step is what determines how fast and how expensive the answer is.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|
A simple definition
An AI model’s life has two main phases. First comes training, or pre-training: the model learns from vast amounts of data and its parameters are set. Then comes inference: you give it an input and it computes an output, without learning anything new. The word comes from reasoning: the model “infers” the most likely continuation from what it has learned.
For a language model, inference happens one token at a time. The model reads your prompt, computes the first piece of the answer, appends it to the text, computes the next one, and so on until it is done.
A concrete example
You ask an assistant to summarize ten pages of meeting notes. The server loads the model into memory, reads the ten pages (the input tokens), then writes the summary word by word (the output tokens). The wait before the first word depends mostly on the length of the document; the total time depends on the length of the summary. If it is a reasoning model, it also produces internal reasoning before answering, which makes inference longer.
Why it matters
Training is a one-off expense; inference is a cost every single time the model is used. That explains the message caps on free plans, API billing by price per million tokens, and the latency you notice. To estimate what an agent making hundreds of calls would cost, try our agent cost calculator.
Common myths
- “The AI learns from every conversation in real time.” No. During inference, the model’s weights don’t move. An assistant’s memory relies on text fed back into the context.
- “Big models are always too slow.” Speed also depends on hardware, server optimization and techniques such as prompt caching or mixture-of-experts architectures.
- “Inference is practically free.” For a single question, close to it. For agents looping over long documents, far from it.
Related terms: latency, price per million tokens, inference server, test-time compute, GPU.
Related terms
- Pre-training · The first training phase of a language model: it learns to predict the next piece of text across a huge corpus, with no specific task.
- Latency · The waiting time between sending a request to an AI model and receiving its answer, from the first word through to the last.
- Price per million tokens · How AI APIs bill usage: one rate per million tokens the model reads as input, and another, usually higher, per million it generates.
- Inference server · Software that loads an AI model into memory and answers requests through an API, while optimizing throughput and GPU memory.
- Test-time compute · The computing power a model spends while answering, for instance to think for longer or to compare several candidate solutions.
- GPU (graphics processing unit) · A chip built to run thousands of calculations in parallel, which became the engine for training and running AI models.
Try it
Full glossary
Frequently asked questions
What's the difference between training and inference?
Training builds the model: its parameters are tuned on huge amounts of data, once, at great expense. Inference uses the model: it happens millions of times, once per question. Think of writing a book versus reading it.
Why do output tokens cost more than input tokens?
The model reads your text in one pass but generates its answer token by token, with each new token requiring a full pass through the network. Providers' price lists reflect this: at Anthropic, for example, output tokens cost several times more than input tokens.
Can I run inference on my own computer?
Yes, with open-weight models and the right software, provided you have enough memory, ideally on a graphics card. Smaller models and quantization make this possible on consumer hardware, with lower performance than the largest models.
Does the model learn from my messages during inference?
Not during inference itself: the model does not change while answering you. A provider may, however, store your conversations and use them for later training, depending on its terms. Check your assistant's privacy settings.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.