VRAM calculator: what hardware to run an LLM locally

How to use it

Pick a model. The 17 models in the first group have a full public architecture: the calculator counts the weights and the KV cache. The 66 models in the second group come from the directory, with the parameter count published on Hugging Face; the KV cache is only counted if you fill in the architecture in the advanced settings. For another model, choose “Custom” and copy the values from its config.json file.

• Quantization: Q4_K_M is the most common local trade-off; Q8_0 is nearly lossless; Q3 and Q2 squeeze in large models at the cost of a clear quality drop.
• Context: the maximum length of conversation or document you plan to use. The KV cache grows in proportion.
• KV cache precision: llama.cpp, Ollama and LM Studio can store the cache in Q8_0 or Q4_0, which halves or quarters it.
• System RAM and number of cards (advanced settings): used for the “runs slowly” verdict and for multi-card setups.

How the calculation works

The memory needed is the sum of three items. The weights: number of parameters × bits per weight ÷ 8. Bits per weight for each quantization format are not round numbers, because “K” formats keep some matrices at higher precision: the tool uses the values measured by the llama.cpp project on Llama 3.1 8B (4.89 bits for Q4_K_M, 8.5 for Q8_0).

The KV cache stores the attention keys and values of every token in the context: 2 × layers × key-value heads × head dimension × context length × bytes per value. Recent models use grouped-query attention (GQA), with few key-value heads, which sharply reduces this item. Finally, an overhead covers compute buffers and the driver context; 10% is a cautious starting point, to adjust for your software.

The verdict compares the total with each device's memory. On a graphics card: runs if everything fits in VRAM; runs slowly if the surplus fits in system RAM (offloaded layers go through the CPU, often five to twenty times slower); otherwise does not run. On a Mac, macOS gives the GPU only part of the unified memory by default: about two thirds up to 36 GB, three quarters above. Beyond that limit and up to 90% of memory, the verdict is “slowly”, because you have to raise the limit by hand and the system then runs short of memory.

A worked example

Llama 3.1 8B in Q4_K_M: 8.03 billion × 4.89 bits ÷ 8 ≈ 4.6 GB of weights (the published GGUF file is 4.58 GiB, which validates the formula). With 8,192 tokens of context in F16: 2 × 32 layers × 8 heads × 128 × 8,192 × 2 bytes = 1 GiB of cache. With the 10% overhead, you need about 6.1 GiB: an 8 GB card is enough. At 128,000 tokens of context, the cache grows to 16 GiB and you need a 24 GB card, or a cache quantized to Q4_0.

Limits of the estimate

• Mixture-of-experts (MoE) models. All experts must be loaded: memory depends on the total parameter count, not active parameters. They do run faster than their size suggests, which makes offloading to RAM more bearable.
• Sliding-window attention. Some models (Gemma 2, for example) limit attention in half of their layers to a short window: their real KV cache is smaller than the one computed here.
• Several cards. Memory adds up, but each card keeps its own overhead and the split is never perfect.
• Memory already in use. The display, browser and other apps often take 0.5 to 2 GB of VRAM or unified memory.
• Speed. “Runs” says nothing about throughput: it mostly depends on the card's memory bandwidth.

To install and choose a local model, see the guide Local LLMs; to understand the formats, Quantization and GGUF.