GGUF: the file format behind local AI models
GGUF is a file format that packs everything needed to run a language model into a single file: its weights, often compressed, and the information required to use them. It is the go-to format for running open models on a personal computer.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|---|
| Origin | 2023 |
| Introduced by | Georgi Gerganov (llama.cpp) |
The definition
GGUF is a file format that holds a complete language model, ready to be loaded locally.
In practice, a GGUF file combines two things: metadata stored as key-value pairs (architecture, tokenizer, context length, quantization type and so on) and the tensors, that is, the model’s parameters. The official specification stresses three goals: single-file deployment, fast loading compatible with memory mapping (mmap), and extensibility, so new information can be added without breaking older files.
The format was introduced in Georgi Gerganov’s llama.cpp project through a contribution merged in August 2023. It replaces the ecosystem’s earlier formats (GGML, GGMF, GGJT). That sets it apart from safetensors, the common Hugging Face format, which stores only tensors: GGUF also carries all the context needed to run the model.
A concrete example
On an open-weights model repository, you will often see a list of files such as model-8B-Q4_K_M.gguf or model-8B-Q8_0.gguf. The suffix names the quantization level. Hugging Face’s documentation describes Q4_K at 4.5 bits per weight: for 8 billion parameters, that gives 8 × 4.5 ÷ 8 = 4.5 GB of weights, versus about 16 GB in 16-bit precision. You pick the file that fits in your VRAM, then load it in llama.cpp, LM Studio or Ollama.
Why it matters
GGUF made local AI practical: one file to download, no configuration to piece together, and quantized variants for every size of machine. It is also an input format for several inference servers, so you can move from a test on a laptop to a local service reachable through an API.
Common misconceptions
- “GGUF is a compression method.” It is a container; it can store 16-bit weights as well as heavily quantized ones.
- “All types are equal.” Hugging Face’s documentation lists some types, such as Q4_0 or Q8_0, as legacy formats, in favor of the K and IQ families.
- “A GGUF file works with any software.” You need an engine that supports the model’s architecture.
Related terms: quantization, VRAM, inference server and inference.
Related terms
- Quantization · A technique that stores a model's weights with fewer bits (8, 4 or even fewer) so it needs far less memory to run.
- VRAM (video memory) · The memory built into a graphics card, where an AI model must fit to run fast; it is often what limits what you can run locally.
- Open-weights model · An AI model whose trained parameters (the weights) can be downloaded, so you can run it on your own hardware.
- Inference server · Software that loads an AI model into memory and answers requests through an API, while optimizing throughput and GPU memory.
- Model parameters · The internal numbers a model tunes during training; how many it has indicates its size, not directly how good it is.
- Inference · The moment a trained model is actually used to produce an output: every question you send to an AI triggers an inference run.
Try it
Full glossary
Frequently asked questions
What do Q4_K, Q5_K or Q8_0 mean in a GGUF file name?
They are quantization types. The number roughly gives the bits per weight: Hugging Face's documentation lists Q4_K at 4.5 bits per weight and Q6_K at about 6.56 bits. The lower the number, the lighter the file, with a growing loss of quality.
Which software can read GGUF files?
llama.cpp, which created the format, plus tools built on top of it. Hugging Face's documentation names LM Studio, GPT4All and Ollama among them.
Can I convert a model to GGUF?
Yes. llama.cpp ships conversion scripts, and Hugging Face offers an online tool, gguf-my-repo, that converts and quantizes a model hosted on its platform.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.