TechCompare LogoTechCompare

How much VRAM does Qwen 2.5 72B need at Q4_K_M? Long-context inference

The KV cache is the hidden cost of long-context models. If you're not actually using the full 128K context, set the context to 8K or 16K in your inference engine - the savings are immediate. Q8 KV is the other obvious lever and almost always worth it.

Qwen 2.5 72B at Q4_K_M with the model's native 128K context needs about 91.6 GB of VRAM. Note how the long context cache of 43 GB exceeds the 40 GB of weights.

By TechCompare · Updated

Total VRAM required
91.6 GB
Qwen 2.5 72B at Q4_K_M
Weights
40.3 GB
72B params
KV cache
42.9 GB
128K tokens, FP16 KV

Calculator

Estimated VRAM required

91.6 GB

72B params at Q4_K_M, 131,072 token context, batch 1, inference.

Weights
40.3 GB
KV cache
42.9 GB
Overhead
8.3 GB

Estimate accuracy: Weights within ~2%. KV cache within ~5% for standard GQA models, ~10% for MLA (DeepSeek). Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

KV cache exceeds model weights: Consider lowering the context length to save on VRAM. Contexts between 8K and 64K are generally more typical for local setups.

Hardware that fits

Apple M3 Ultra 128GB
Unified
96 GB
95% used
H200 141GB
Datacenter
141 GB
65% used
Apple M3 Ultra 192GB
Unified
144 GB
64% used

Just barely too small

A100 80GB
Datacenter
80 GB
short by 11.6 GB
H100 80GB
Datacenter
80 GB
short by 11.6 GB
Same 72B @ Q4_K_M, 131,072 context scenario, pre-loaded. Change the quant, shrink the context, or try a different preset.
Edit in the full tool

How this is calculated

At 128K with FP16 KV, the cache for this model is around 43 GB on its own. Weights take 40.3 GB, and overhead is 8.3 GB, totaling 91.6 GB.

Verdict

At 128K with FP16 KV the cache alone runs 43 GB, which actually exceeds the 40.3 GB of Q4 weights, with 8.3 GB of overhead bringing the total to 91.6 GB. That's the inversion this entry exists to flag: for 70B-class models at native context, the cache dominates the budget. Drop the context to 16K and the cache falls to roughly 5.4 GB, which compresses the total to about 54 GB and makes a 24 GB card viable with reduced context rather than requiring two pooled cards. Q8 KV quantization halves the cache row (43 GB to 21.4 GB) without materially affecting inference quality, and is the lever to pull first when you need more headroom than the context cap grants.

More Qwen scenarios

DeepSeek V4 Pro 1.6T (MoE) at Q4_K_M
DeepSeek V4 Pro 1.6T at Q4_K_M with the full 1M-token context needs about 1012 GB of VRAM with every expert resident - this is the real shape of the model and the number to plan a deployment against.
View details ➜
Llama 4 Scout (17B/109B) at Q4_K_M
Llama 4 Scout at Q4_K_M with its native 10M context needs about 2231 GB of VRAM with all 109B params resident - that's the number you size hardware against.
View details ➜
gpt-oss 20B (MoE) at Q4_K_M
gpt-oss 20B at Q4_K_M with native 128K context needs about 19.4 GB of VRAM with all experts resident, dropping to roughly 9.3 GB with active-only weight loading.
View details ➜

Related guides

BPE tokenization explained
How tokenizers chunk text into the units VRAM serves.
View guide ➜

Frequently asked questions

Why is the VRAM higher than Llama 3.1 70B at the same quant?
Qwen 2.5 72B is slightly larger (72B vs 70B params) and we are running at its native context. At matched context and parameters they are within 5% of each other.
Does Q8 KV cache hurt quality?
Not measurably for inference. Q8 KV is widely used in production llama.cpp builds with no observable change in output. Use it freely as a memory-saving lever.
Why does the VRAM look higher than Llama 3.1 70B at the same Q4?
Because Qwen 2.5 72B uses a larger model architecture (more layers, more attention heads, and a slightly larger hidden dimension) which makes the KV cache bigger at the same context window. Plus Qwen 2.5's Q8 KV cache option is more memory-expensive but preserves quality. Plan for roughly 45 GB resident at standard Q4 + FP16 KV cache at 32K context, similar to Llama 70B.
Why do two models with the same parameter count need different amounts of VRAM?
Because parameter count covers only the weights. The KV cache depends on layers, attention heads, and head dimension, all architecture choices that Qwen 2.5 pushes higher than Llama at a similar parameter scale. The bigger the KV cache, the more the context length multiplies total memory. Two 70B-class models can honestly differ by several gigabytes at long context for this reason alone.

Related tools

RAM Latency Calculator

Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.

Use tool ➜

Power Cost Estimator

Estimate annual electricity costs for your PC, Server, or TV.

Use tool ➜

Data Transfer Calculator

Estimate transfer times for files over USB, WiFi, Ethernet, and more.

Use tool ➜

Memory and Storage Latency Visualizer

Visualize the massive speed difference between CPU cache, RAM, and storage.

Use tool ➜