TechCompare LogoTechCompare
Showing centimeters. Press to switch to inches.Showing inches. Press to switch to centimeters.

AI

KV cache

The attention keys and values a model stores for every token it has already read, so it doesn't recompute them for each new token. It grows with the length of the conversation and can take more memory than the model weights at long context.

What the cache holds

At every layer, attention gives each token a key vector and a value vector. To write the next token, the model compares the new token against the keys of all earlier tokens and blends their values. Recomputing those for the whole conversation at every step would waste work, so the runtime keeps them. That store is the KV cache.

It belongs to the conversation, not the model. It starts empty, gains one token's worth of entries for each token read or written, and goes away when the conversation ends. Run several conversations at once and each one gets its own.

How big it gets

The size follows a simple formula:

cache bytes = 2 x layers x KV heads x head size x bytes per value x tokens

The 2 is for keys and values. Llama 3.1 8B has 32 layers, 8 KV heads and a head size of 128, and an FP16 cache uses 2 bytes per value. The calculator gives these sizes for the cache alone:

  • 8K tokens: 1.1 GB
  • 32K tokens: 4.3 GB
  • 128K tokens: 17.2 GB

The cache grows in a straight line, so doubling the context doubles it. At 128K it's bigger than the 4.9 GB of Q4_K_M weights, which is why a model that loads fine at short context can run out of VRAM at long context.

How models shrink it

  • Fewer KV heads. Grouped query attention shares each key and value across a group of query heads, using a number of KV heads between one and the full count. Llama 3.1 8B has 32 attention heads and 8 KV heads. With a KV head for every attention head, the same 128K cache would take 68.8 GB.
  • Compressed attention. DeepSeek-V3 uses multi head latent attention, which stores a compressed form of the keys and values. In the calculator, its 128K cache is 10 GB, against 43 GB for Llama 3.1 70B. The attention design counts for as much as the model's size here.
  • A quantized cache. Storing the cache at 8 bits instead of 16 halves it. Llama 3.1 8B at 128K goes from 17.2 GB to 8.6 GB.
  • A shorter context. Cap the window at what the task needs.

Not the same as prompt caching

Prompt caching is a provider feature that keeps the processed state of a repeated prompt prefix between requests, so you pay less to send it again. OpenAI's docs say that cache stores the KV tensors for the prefix. The KV cache on this page is the working memory of one running conversation, and prompt caching reuses that same kind of data across requests.

Work it out yourselfOpen the LLM VRAM Calculator

Read next

DeepSeek V3 at 128K context Llama 3.1 70B at Q4_K_M Context windows explained

Related terms

Context windowThe most text a language model can handle in one request, counted in tokens. The prompt, the conversation so far and the model's reply all share it, so a long input leaves less room for the answer.VRAMThe memory on a graphics card, next to the GPU. To run a large language model on a GPU, its weights, its KV cache and some runtime overhead all have to fit in VRAM, so VRAM decides which models and context lengths a card can handle at full speed.Prompt cachingAn API feature that keeps the processed start of your prompt for a short time, so a later request with the same start isn't computed or billed at the full input rate again. It pays off when many requests share a long system prompt, tool list or document.QuantizationStoring a model's weights with fewer bits per number, like 4 or 8 bits instead of 16. A quantized model takes far less VRAM and usually generates tokens faster, in exchange for a small drop in quality.

Sources

Back to the glossary