VRAM: the memory that decides which AI models you can run
VRAM is the memory built directly into a graphics card. For an AI model to run fast on a GPU, its weights and working memory must fit there, so VRAM, more than raw power, usually sets the size of the models you can run locally.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
The definition
VRAM is the graphics card’s own memory, separate from your computer’s main memory (RAM).
A GPU computes very fast, but only on data sitting in its own memory. For AI, that means the model’s weights and, during generation, the KV cache and intermediate results. VRAM is described by two numbers: capacity (in gigabytes), which decides what fits, and bandwidth, which affects speed, because generating each token means reading through a large share of the weights again.
How to estimate what you need
Hugging Face’s documentation offers a simple rule: a model with X billion parameters needs about 4 × X GB in 32-bit precision and 2 × X GB in 16-bit precision, for the weights alone. On top of that, add:
- the KV cache, which grows with context length and the number of simultaneous requests;
- headroom for intermediate calculations and the software itself.
Worked example. An 8-billion-parameter model needs about 16 GB in 16-bit precision. With the Q4_K quantization type from the GGUF format, which Hugging Face describes at 4.5 bits per weight, the math gives 8 × 4.5 ÷ 8 = 4.5 GB for the weights. That brings the model within reach of a consumer graphics card, as long as you leave room for the context.
Why it matters
VRAM is the most expensive and most constrained resource in local AI and in data centers alike. The same documentation notes that models such as Llama 2 70B (around 140 GB in 16-bit) exceed the 80 GB of A100 and H100 cards and must be split across several GPUs. Techniques such as FlashAttention exist precisely to limit trips to this memory, which is slower than the chip’s on-board memory.
Common myths
- “Lots of RAM is enough.” System RAM can hold a model, but computing from it is much slower than from VRAM, except on machines with shared memory.
- “If the weights fit, everything fits.” A long context can add several gigabytes of cache.
- “More VRAM makes the model smarter.” It lets you load a bigger or less compressed model; it does not change a given model.
Related terms: GPU, quantization, GGUF format and inference.
Related terms
- GPU (graphics processing unit) · A chip built to run thousands of calculations in parallel, which became the engine for training and running AI models.
- Quantization · A technique that stores a model's weights with fewer bits (8, 4 or even fewer) so it needs far less memory to run.
- KV cache · Memory where a model keeps the work already done on previous tokens so it does not have to redo it for every new word.
- Model parameters · The internal numbers a model tunes during training; how many it has indicates its size, not directly how good it is.
- GGUF (model file format) · A single-file format bundling a model's weights, often quantized, with its metadata, so it can run locally with llama.cpp and related tools.
- Inference · The moment a trained model is actually used to produce an output: every question you send to an AI triggers an inference run.
Try it
Full glossary
Frequently asked questions
How much VRAM do I need for a 7B or 8B model?
In 16-bit precision, count roughly 2 GB per billion parameters, so around 15 GB for the weights alone. Quantized to about 4 bits, the same model drops to a few gigabytes, plus room for the KV cache and some headroom.
What happens if the model does not fit in VRAM?
Some software, such as llama.cpp, can keep part of the layers in regular system RAM and run them on the CPU. The model still works, but much more slowly, because transfers between the card and the rest of the computer become the bottleneck.
Is VRAM also used for training?
Yes, and training needs far more: besides the weights, it stores gradients and optimizer state. That is why methods such as LoRA and QLoRA, which cut those needs, are so widely used.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.