Quantization: fitting an AI model into less memory
Quantization stores the numbers inside an AI model in smaller boxes, for example 4 bits instead of 16, so the model needs much less memory. It is the technique that lets you run an open model on a personal computer.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
The one-line idea, then the details
Quantization means writing a model’s numbers with fewer digits so the model takes up less space.
A large language model is a huge table of decimal numbers called parameters, or weights. They are usually stored in 16 bits (bfloat16 or float16), which is 2 bytes per parameter. Quantizing converts them to a shorter format: 8 bits, 4 bits, sometimes 3 or 2. There are two broad families:
- post-training quantization (PTQ), applied to a finished model without retraining it; GPTQ, AWQ and the llama.cpp formats belong here;
- quantization-aware training (QAT), where the model learns during training to cope with lower precision.
Modern methods work in blocks: a small group of weights shares one scale factor, which keeps rounding error in check. They also handle outliers separately, because a handful of extreme values can wreck the whole computation if they are rounded carelessly.
A worked example
Hugging Face’s documentation offers a simple rule of thumb: in 16-bit precision, a model with X billion parameters needs roughly 2 × X GB of VRAM just for its weights. A 15-billion-parameter model therefore needs around 30 GB, more than a typical consumer graphics card offers. In the same guide, that model drops to about 15 GB in 8-bit and just over 9 GB in 4-bit, with almost identical output on the example shown.
Why it matters
Quantization is what made local AI practical. Two 2022 papers set the pace: LLM.int8() by Tim Dettmers and co-authors, which halves inference memory with no measured loss in their experiments, and GPTQ, which goes down to 3 or 4 bits per weight. Many open-weight models are now distributed in quantized versions, often in the GGUF format. For an API provider, less memory per model also means more requests served per machine.
Common myths
- “Quantized means broken.” The reference papers report negligible loss at 8 bits and small loss at 4 bits. Below that, degradation becomes noticeable, and each model reacts differently.
- “It is always faster.” The main gain is memory; speed depends on compute kernels and the card itself.
- “Only weights can be quantized.” The KV cache and activations can be quantized too.
- “You cannot fine-tune a quantized model.” QLoRA (2023) combines 4-bit quantization with LoRA precisely to fine-tune large models on a single GPU.
Related terms: VRAM, GGUF format, LoRA, KV cache and inference.
Related terms
- VRAM (video memory) · The memory built into a graphics card, where an AI model must fit to run fast; it is often what limits what you can run locally.
- GGUF (model file format) · A single-file format bundling a model's weights, often quantized, with its metadata, so it can run locally with llama.cpp and related tools.
- Model parameters · The internal numbers a model tunes during training; how many it has indicates its size, not directly how good it is.
- Open-weights model · An AI model whose trained parameters (the weights) can be downloaded, so you can run it on your own hardware.
- LoRA (Low-Rank Adaptation) · A fine-tuning method that freezes the model and trains only small added matrices, making it cheap to adapt a large language model.
- Inference · The moment a trained model is actually used to produce an output: every question you send to an AI triggers an inference run.
Try it
Full glossary
Frequently asked questions
What do Q4, Q5 and Q8 mean in a model file name?
The number roughly indicates how many bits are used per weight. Q8 stays very close to the original but is about twice the size of Q4; Q4 is the most common trade-off for local use. Suffixes such as K_M refer to llama.cpp's block-wise quantization methods.
Does quantization make a model faster?
Not necessarily. Its first effect is lower memory use. Speed depends on your hardware and software, and Hugging Face's documentation even notes that some methods slow inference slightly because weights are converted back on the fly.
Can I quantize any model?
You need access to the weights, so it applies to open-weight models. Closed models served through an API are optimized by their provider, which usually does not disclose the precision it uses.
How do I know whether the quality loss is acceptable?
Compare the quantized model with the original on your own tasks. Small models and aggressive settings below 4 bits are the most sensitive.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.