One budget for everything
The window isn't just your question. It holds the system prompt, the whole conversation so far, any tool definitions or attached files, and the model's own reply, thinking included. The Claude docs describe it the same way: input and output both count.
That makes the arithmetic simple and a little unforgiving. A 128,000 token window holds around 96,000 words of English. Paste in a 100,000 token document and 28K tokens remain for everything else. A long chat resends its history with each turn, so it fills the window even when every message is short.
What a bigger window costs in VRAM
When you run a model yourself, every token in the window needs its keys and values stored in the KV cache, so memory grows with the window. For Llama 3.1 8B at Q4_K_M, the calculator's totals are:
- 8K tokens: 6.6 GB
- 32K tokens: 10.1 GB
- 128K tokens: 24.3 GB
At 128K the cache alone is 17.2 GB, bigger than the 4.9 GB of weights. A model's advertised window is a ceiling, not a promise that your GPU can reach it.
A long window isn't a good memory
Models don't use every part of a long prompt equally well. Liu and colleagues found that accuracy is highest when the relevant information sits at the start or the end of the context, and it drops when the information is in the middle, even for models built for long inputs. Anthropic's docs make the broader point: more context isn't automatically better, because accuracy and recall can degrade as the token count grows, which they call context rot.
So curating what goes in matters as much as how much fits. Put instructions and the key documents first or last, cut anything the model doesn't need, and retrieve the relevant passages instead of pasting whole files.
When you hit the limit
What happens at the limit depends on the API or app. Some reject the request with an error, some drop the oldest turns, and chat apps often summarize earlier messages. Check how yours behaves before you rely on it, and watch the token counts the provider returns.
To stretch a window, summarize old turns, retrieve only what's relevant, and use prompt caching for text that repeats. Caching lowers the price of that repeated text, but the text still takes up room in the window.