TechCompare LogoTechCompare

How much VRAM does Qwen 2.5 72B need at Q4_K_M? Long-context inference

The KV cache is the hidden cost of long-context models. If you're not actually using the full 128K context, set the context to 8K or 16K in your inference engine - the savings are immediate. Q8 KV is the other obvious lever and almost always worth it.

Qwen 2.5 72B at Q4_K_M with the model's native 128K context needs about 91.6 GB of VRAM. Note how the long context cache of 43 GB exceeds the 40 GB of weights.

By TechCompare · Updated

Total VRAM required
91.6 GB
Qwen 2.5 72B at Q4_K_M
Weights
40.3 GB
72B params
KV cache
42.9 GB
128K tokens, FP16 KV

Calculator

Estimated VRAM required

91.6 GB

72B params at Q4_K_M, 131,072 token context, batch 1, inference.

Weights
40.3 GB
KV cache
42.9 GB
Overhead
8.3 GB

Estimate accuracy: Weights within ~2%. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

KV cache exceeds model weights: Consider lowering the context length to save on VRAM. Contexts between 8K and 64K are generally more typical for local setups.

Hardware that fits

Apple entries show installed unified memory separately from a model budget based on an 8 GB reserve. That budget is a planning heuristic, not a fixed hardware limit.

H200 141GB
Datacenter
141 GB
65% used
MI300X
Datacenter
192 GB
48% used
Apple M3 Ultra 256GB
Unified
~248 GB
37% of estimate

Just barely too small

Apple M3 Ultra 96GB
Unified
~88 GB
short by 3.6 GB
A100 80GB
Datacenter
80 GB
short by 11.6 GB
H100 80GB
Datacenter
80 GB
short by 11.6 GB
Same 72B @ Q4_K_M, 131,072 context scenario, pre-loaded. Change the quant, shrink the context, or try a different preset.
Edit in the full tool

How this is calculated

At 128K with FP16 KV, the cache for this model is around 43 GB on its own. Weights take 40.3 GB, and overhead is 8.3 GB, totaling 91.6 GB.

Verdict

At 128K with FP16 KV the cache alone runs 43 GB, which actually exceeds the 40.3 GB of Q4 weights, with 8.3 GB of overhead bringing the total to 91.6 GB. That's the inversion this entry exists to flag: for 70B-class models at native context, the cache dominates the budget. Drop the context to 16K and the cache falls to roughly 5.4 GB, which compresses the total to about 54 GB and makes a 24 GB card viable with reduced context rather than requiring two pooled cards. Q8 KV quantization halves the cache row (43 GB to 21.4 GB) without materially affecting inference quality, and is the lever to pull first when you need more headroom than the context cap grants.

More Qwen scenarios

MiMo-V2.6-Pro at Q4_K_M
Budget a 1.02T expert pool, hybrid attention, and a 1M text context.
View details ➜
MiMo-V2.6-Flash at Q4_K_M
309B total weights and separate global and sliding cache widths.
View details ➜
GLM-5.3-Flash at Q4_K_M
Model the 34 KDA layers, 11 sparse layers, and indexer storage.
View details ➜

Related guides

BPE tokenization explained
How tokenizers chunk text into the units VRAM serves.
View guide ➜

Frequently asked questions

Why is the VRAM higher than Llama 3.1 70B at the same quant?
Qwen 2.5 72B is slightly larger (72B vs 70B params) and we are running at its native context. At matched context and parameters they are within 5% of each other.
Does Q8 KV cache hurt quality?
Not measurably for inference. Q8 KV is widely used in production llama.cpp builds with no observable change in output. Use it freely as a memory-saving lever.
Why does the VRAM look higher than Llama 3.1 70B at the same Q4?
Because Qwen 2.5 72B uses a larger model architecture (more layers, more attention heads, and a slightly larger hidden dimension) which makes the KV cache bigger at the same context window. Plus Qwen 2.5's Q8 KV cache option is more memory-expensive but preserves quality. Plan for roughly 45 GB resident at standard Q4 + FP16 KV cache at 32K context, similar to Llama 70B.
Why do two models with the same parameter count need different amounts of VRAM?
Because parameter count covers only the weights. The KV cache depends on layers, attention heads, and head dimension, all architecture choices that Qwen 2.5 pushes higher than Llama at a similar parameter scale. The bigger the KV cache, the more the context length multiplies total memory. Two 70B-class models can honestly differ by several gigabytes at long context for this reason alone.

Related tools

RAM Latency Calculator

Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.

Use tool ➜

Power Cost Estimator

Estimate annual electricity costs for your PC, Server, or TV.

Use tool ➜

Data Transfer Calculator

Estimate transfer times for files over USB, WiFi, Ethernet, and more.

Use tool ➜

Memory and Storage Throughput Visualizer

Visualize the massive speed difference between CPU cache, RAM, and storage.

Use tool ➜