TechCompare LogoTechCompare

How much VRAM does Llama 3.1 8B need at 128K context? Long-context cost

If you're not actually using 128K of context, don't allocate for it. Inference engines like llama.cpp and vLLM let you cap context at runtime. Setting it to 16K or 32K instead of 128K saves more VRAM than any quantization change you could make.

Llama 3.1 8B at Q4_K_M with the full 128K context needs about 24 GB of VRAM - nearly 4x the 6 GB it uses at 8K context, even though the model itself is unchanged. The KV cache balloons to ~17 GB on its own, exceeding the weight memory and dominating total usage.

By TechCompare · Updated

Total VRAM required
23.8 GB
Llama 3.1 8B at Q4_K_M
Weights
4.5 GB
8B params
KV cache
17.2 GB
128K tokens, FP16 KV

Calculator

Estimated VRAM required

23.8 GB

8B params at Q4_K_M, 131,072 token context, batch 1, inference.

Weights
4.5 GB
KV cache
17.2 GB
Overhead
2.2 GB

Estimate accuracy: Weights within ~2%. KV cache within ~5% for standard GQA models, ~10% for MLA (DeepSeek). Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

KV cache exceeds model weights: Consider lowering the context length to save on VRAM. Contexts between 8K and 64K are generally more typical for local setups.

Hardware that fits

RTX 3090
Consumer
24 GB
99% used
RTX 5090
Consumer
32 GB
74% used
A100 40GB
Datacenter
40 GB
60% used
Apple M3 Max 64GB
Unified
48 GB
50% used
Same 8B @ Q4_K_M, 131,072 context scenario, pre-loaded. Change the quant, shrink the context, or try a different preset.
Edit in the full tool

How this is calculated

KV cache scales linearly with context length and the cost is independent of quantization (KV cache is its own dtype). For Llama 3.1 8B (32 layers, 4096 hidden), each token in context uses 4 KB of FP16 KV memory (with 8 KV heads), so 128K tokens is roughly 17 GB total memory across all layers. Switching to Q8 KV halves that to 8.5 GB, and reducing the working context to what you actually need is by far the biggest lever.

Verdict

The math is stark: at 8K context Llama 3.1 8B at Q4_K_M needs about 6 GB total (4.5 GB of weights plus a 1 GB cache plus 2 GB overhead), and at 128K context the same model needs 24 GB because the FP16 KV cache alone balloons to about 17 GB (8 KV heads at head_dim 128 across 32 layers times 128K tokens times 2 bytes). That is a 4x budget swing from a runtime flag, larger than any Q4-vs-Q8-vs-FP16 quantization change you could make at the weight layer. Two operational levers close the gap: cap context in your inference engine to what you actually need (16K or 32K is usually plenty), and switch to Q8 KV quantization which halves the cache row to about 8.5 GB and is widely treated as quality-neutral in production.

More Llama scenarios

DeepSeek V4 Pro 1.6T (MoE) at Q4_K_M
DeepSeek V4 Pro 1.6T at Q4_K_M with the full 1M-token context needs about 1012 GB of VRAM with every expert resident - this is the real shape of the model and the number to plan a deployment against.
View details ➜
Llama 4 Scout (17B/109B) at Q4_K_M
Llama 4 Scout at Q4_K_M with its native 10M context needs about 2231 GB of VRAM with all 109B params resident - that's the number you size hardware against.
View details ➜
gpt-oss 20B (MoE) at Q4_K_M
gpt-oss 20B at Q4_K_M with native 128K context needs about 19.4 GB of VRAM with all experts resident, dropping to roughly 9.3 GB with active-only weight loading.
View details ➜

Related guides

BPE tokenization explained
How tokenizers chunk text into the units VRAM serves.
View guide ➜

Frequently asked questions

Why does long context cost so much VRAM?
Every token in the context window stores its key and value vectors, one per attention head per layer. At 128K tokens that's hundreds of millions of values cached for the duration of the request.
Does prompt caching help with long contexts?
Yes, prefix caching reuses the KV cache across requests that share a prompt prefix. It doesn't reduce memory for one request but dramatically reduces re-computation across many.
Why does long context cost so much VRAM?
KV cache scales linearly with context length. At 128K context for Llama 3.1 8B with 8 KV heads at head_dim 128 across 32 layers, the FP16 KV cache alone is about 4 GB on top of the 4 GB weight footprint. Going from 8K to 128K context multiplies the KV cache by 16x, which is why long-context inference often needs Q8 (1 byte/value) KV cache quantization to fit.
What is Q8 KV cache quantization and when should I enable it?
Q8 stores the KV cache in 8 bits per value instead of 16, halving the cache's VRAM while keeping quality loss below the perceptible floor on most tasks. Enable it when context length is the binding constraint: a 128K conversation that won't fit with FP16 KV often fits comfortably at Q8. Skip it when you're memory-rich, since FP16 is the default for a reason and Q8 buys nothing if length isn't your bottleneck.

Related tools

RAM Latency Calculator

Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.

Use tool ➜

Power Cost Estimator

Estimate annual electricity costs for your PC, Server, or TV.

Use tool ➜

Data Transfer Calculator

Estimate transfer times for files over USB, WiFi, Ethernet, and more.

Use tool ➜

Memory and Storage Latency Visualizer

Visualize the massive speed difference between CPU cache, RAM, and storage.

Use tool ➜