What the names in a model file mean
A model's weights are billions of numbers, and at FP16 each one takes 16 bits. Quantization rounds them to fewer bits and stores a scale alongside, so the file shrinks. Local runtimes like llama.cpp use names such as Q8_0 and Q4_K_M for this. The digit after the Q is the nominal bit width, and the real average comes out higher. The llama.cpp README measures it for Llama 3.1 8B:
- F16: 16.0005 bits per weight, 14.96 GiB
- Q8_0: 8.5008 bits per weight, 7.95 GiB
- Q4_K_M: 4.8944 bits per weight, 4.58 GiB
- Q2_K: 3.1593 bits per weight, 2.95 GiB
Other methods follow the same idea. GPTQ compresses a trained model to 3 or 4 bits per weight, and AWQ protects the 1% of weights that matter most from rounding error.
What it saves
Weights take parameters times bytes per parameter, so the saving is easy to see. An 8B model needs 16.0 GB at FP16, 8.5 GB at Q8_0 and 4.9 GB at Q4_K_M. A 70B model goes from 140 GB at FP16 to 43 GB at Q4_K_M. The FP16 version doesn't fit on one 80 GB H100. With an 8K context the Q4_K_M version needs 50 GB, just over a single 48 GB workstation card.
Quantization only shrinks the weights. The KV cache and the runtime's overhead stay as they were unless you also quantize the cache, so a model that fits on paper can still run out of VRAM once the context grows. The LLM VRAM Calculator adds all three. The calculator's 0.61 bytes per parameter for Q4_K_M comes from real files like these: the README's 4.8944 bits is 0.612 bytes per weight.
What it costs in quality, and what it gains in speed
Rounding adds error, and how much it hurts depends on the bits, the method and the model. The GPTQ paper reports negligible accuracy loss at 3 or 4 bits per weight for large GPT models. Dettmers and Zettlemoyer, who compared bit widths across many models, found 4-bit precision almost universally optimal for the trade between total model bits and zero-shot accuracy. The llama.cpp README says accuracy loss is usually measured with perplexity or KL divergence, and that an importance matrix (imatrix) file can help minimize it. Those are benchmark averages, so test a quant on your own prompts before you commit to it.
Smaller files also generate text faster, because each new token needs a pass over the whole model. In the README's Llama 3.1 8B test, text generation ran at 29.17 tokens per second at F16, 50.93 at Q8_0 and 71.93 at Q4_K_M. That puts Q4_K_M at 2.5 times the F16 rate. Reading the prompt barely changed: about 923 tokens per second at F16 and 822 at Q4_K_M, so the gain shows up in generation.
Picking one
- Q8_0 loses the least of the common quants, since it keeps the most bits, and it still saves a lot of memory. Use it when it fits.
- Q4_K_M is the usual starting point for local use. It sits at the 4 bit point that Dettmers and Zettlemoyer found best for size against accuracy.
- Q3 and Q2 variants make sense when nothing larger fits. Test them on your own work first.
- If Q4_K_M already fits, a longer context is often a better use of the spare VRAM than a lower quant.