How a cache hit happens
When a request arrives, the provider can look for earlier work on the same opening text. If the start of your prompt matches a cached prefix exactly, it reuses the stored result instead of processing those tokens again. OpenAI's docs say what's stored is the KV tensors, and that a hit needs the entire prefix to match. Anthropic builds the prompt in the order tools, system prompt, messages, and a hit needs those segments to be 100% identical.
One changed token ends the match at that point, so layout matters:
- Put the static parts first: system prompt, tool definitions, reference documents and examples.
- Put what changes last: the user's new message, timestamps, IDs and retrieved snippets.
- Don't reorder or edit the earlier parts between requests.
Some providers cache by default. OpenAI's caching is automatic, and Google's implicit caching is on by default for Gemini 2.5 and newer models. Anthropic uses explicit breakpoints: you mark where the cached prefix ends with cache_control.
How the bill changes
Input tokens fall into up to three kinds on a cached request:
- Regular input: tokens that weren't in the cache, billed at the normal input rate.
- Cache reads: tokens served from a hit, billed at a lower cached rate.
- Cache writes: tokens stored for later. Some providers charge a premium for these and others charge nothing extra.
Output tokens are billed as usual. The usage data in each response shows what happened. OpenAI reports cached_tokens, Anthropic reports cache_read_input_tokens and cache_creation_input_tokens, and Gemini reports usage.total_cached_tokens.
Rates and discounts differ by provider and model and change often, so this page doesn't list them. The LLM Pricing Calculator shows each model's current cached input price and lets you set how much of your input is cached.
A worked example
Say an assistant starts every request with a 20,000 token prefix (instructions, tool definitions and a reference document) and adds a 1,000 token question. About 95% of each request's input is the repeated prefix. Over 500 requests that prefix alone is 10,000,000 input tokens. Without caching, every one is billed at the full input rate. With caching, the first request builds the cache, and later requests inside its lifetime read the prefix at the cached rate.
The usual way to lose all of it is a changing first line. Put a timestamp at the top of the system prompt and every request starts with a new token, so nothing ever matches. Check the cached token counts in the usage data instead of assuming hits. If they read zero, a changed prefix or a prompt below the minimum length is the usual cause.
Limits to check
- Minimum length. Providers set a minimum prompt length that varies by model, and shorter prompts aren't cached.
- Lifetime. Cache entries expire when unused, often within minutes, and a hit usually renews the clock. Some providers offer a longer option.
- Rate limits. On OpenAI, cached tokens still count toward your tokens per minute limit.
- Context window. Cached text still fills the window. Caching changes the price, not the token count.