Inference server: the software that serves an AI model
An inference server is the program that keeps an AI model loaded in memory and answers the requests sent to it, much like a web server answers visitors. Its goal is to serve as many requests as possible on the same GPU without wasting memory.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
The definition
An inference server is software that runs a model continuously and makes it available through an API.
Inference means using an already trained model to produce an answer. An inference server turns that step into a service: it loads the weights onto one or more GPUs, receives HTTP requests, schedules their execution and streams tokens back as they are generated. Most now expose an API compatible with OpenAI’s, so you can swap a provider for a self-hosted model without rewriting the application.
Examples and key techniques
Among the best-known open projects:
- vLLM, which started at UC Berkeley’s Sky Computing Lab, is built on PagedAttention, described in a 2023 paper: the KV cache is split into blocks, like an operating system’s virtual memory, which limits waste. The authors report 2 to 4 times higher throughput than the systems they compared against.
- llama-server, the llama.cpp server, reads files in the GGUF format and offers OpenAI-compatible endpoints and a web interface.
- Ollama, designed for local use, supports GGUF models.
These engines share common techniques: continuous batching, which adds new requests to a running batch instead of waiting for it to finish; weight quantization (GPTQ, AWQ, FP8 or GGUF, depending on the engine); and speculative decoding.
Why it matters
The cost of an AI service depends on how many requests a GPU can handle. Because generation is mostly limited by memory bandwidth, a good inference server batches requests and manages memory carefully to process more of them on the same hardware. It is also what makes it possible to host an open model on your own infrastructure, for privacy or cost-control reasons.
Common misconceptions
- “The model does everything; the server makes no difference.” With the same weights, the choice of engine and its settings strongly affects throughput and latency.
- “OpenAI-compatible means identical.” Compatibility covers the main endpoints, not always every option.
- “You need a data-center GPU.” llama.cpp and Ollama run on consumer machines with quantized models.
Related terms: inference, KV cache, GGUF and speculative decoding.
Related terms
- Inference · The moment a trained model is actually used to produce an output: every question you send to an AI triggers an inference run.
- KV cache · Memory where a model keeps the work already done on previous tokens so it does not have to redo it for every new word.
- GPU (graphics processing unit) · A chip built to run thousands of calculations in parallel, which became the engine for training and running AI models.
- GGUF (model file format) · A single-file format bundling a model's weights, often quantized, with its metadata, so it can run locally with llama.cpp and related tools.
- Speculative decoding · A speed-up technique where a small model drafts several tokens ahead and the large model checks them all in a single pass.
- Quantization · A technique that stores a model's weights with fewer bits (8, 4 or even fewer) so it needs far less memory to run.
Try it
Full glossary
Frequently asked questions
How is an inference server different from a library such as Transformers?
A library lets you run a model inside a script. An inference server adds a network API, handling of many concurrent requests and memory optimizations, for continuous use by applications.
Is Ollama an inference server?
Yes, for local use: its documentation describes an API served by default on localhost, port 11434, with an endpoint compatible with the OpenAI API.
Is Text Generation Inference (TGI) still recommended?
Its documentation states that it has moved to maintenance mode and points to other engines, including vLLM, SGLang, llama.cpp and MLX.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.