TechCompare LogoTechCompare
Showing centimeters. Press to switch to inches.Showing inches. Press to switch to centimeters.

AI

VRAM

The memory on a graphics card, next to the GPU. To run a large language model on a GPU, its weights, its KV cache and some runtime overhead all have to fit in VRAM, so VRAM decides which models and context lengths a card can handle at full speed.

What VRAM is

VRAM (video RAM) is the memory built onto a graphics card, right next to the GPU. Games fill it with textures and frame buffers. For AI work it holds the model. Gaming cards use GDDR memory, and data center cards like the NVIDIA H100 use HBM, which is stacked beside the chip for even more bandwidth.

It's separate from your system RAM. The GPU reads its own VRAM many times faster than it can pull data from system memory over PCIe, so a model has to fit in VRAM to run at full speed. That makes the amount of VRAM the first number to check: 8 GB on many mainstream cards, 24 GB on an RTX 4090, 32 GB on an RTX 5090 and 80 GB on an H100.

What fills VRAM when you run an LLM

Three things share the space:

  • Weights: every parameter of the model, stored at some precision. At FP16 each one takes 2 bytes. Quantized to Q4_K_M it takes about 0.61 bytes.
  • KV cache: the attention keys and values for every token in the conversation so far. It grows with the context length and the batch size.
  • Overhead: the runtime itself, the CUDA context and working buffers.

The weights are easy to estimate: parameters times bytes per parameter. An 8B model takes 16 GB at FP16 and 4.9 GB at Q4_K_M. The KV cache depends on the model's layers, attention heads and head size, which is where a calculator helps.

A worked example

Take Llama 3.1 8B at Q4_K_M. With an 8K context it needs about 6.6 GB in total. That fits on an 8 GB card. Stretch the context to 128K and the total climbs to 24.3 GB. Most of that is KV cache: 17.2 GB of it. That's past a 16 GB card and just over a 24 GB one.

Llama 3.1 70B is a different class. At Q4_K_M with an 8K context it needs about 50 GB. That's a little more than two 24 GB cards or one 48 GB workstation card hold, so it takes a third card, an 80 GB card or offloading some layers to system RAM.

The LLM VRAM Calculator works this out for any model, quantization, context length and batch size, and tells you how many GPUs the total takes.

When the model doesn't fit

You have four ways out. Pick a smaller quantization, like Q4_K_M instead of Q8_0. Shorten the context, which shrinks the KV cache. Split the model across two or more GPUs, which tools like llama.cpp and vLLM do for you. Or offload some layers to system RAM, which works but runs much slower, since every token has to wait for the layers that live outside VRAM.

A model that just barely fits can still fail once the context fills up, because the KV cache keeps growing as the conversation gets longer. Leave some headroom.

Work it out yourselfOpen the LLM VRAM Calculator

Read next

How much VRAM does Llama 3.1 8B need? How much VRAM does Llama 3.1 70B need? Llama 3.1 8B at 128K context Qwen 2.5 32B on a 24 GB GPU

Related terms

QuantizationStoring a model's weights with fewer bits per number, like 4 or 8 bits instead of 16. A quantized model takes far less VRAM and usually generates tokens faster, in exchange for a small drop in quality.KV cacheThe attention keys and values a model stores for every token it has already read, so it doesn't recompute them for each new token. It grows with the length of the conversation and can take more memory than the model weights at long context.Context windowThe most text a language model can handle in one request, counted in tokens. The prompt, the conversation so far and the model's reply all share it, so a long input leaves less room for the answer.Parameter countThe number of learned weights in a model, written as 8B, 70B or 405B for billions. It sets how big the model file is and how much memory it needs, but it doesn't tell you how good the model is.

Sources

Back to the glossary