What is a context window? How LLMs remember your conversation

How this is calculated

The context window includes both input and output tokens. If a model has a 128K context window and you send a 100K token document, you only have 28K tokens left for the model's response and any follow-up messages. The window is shared across the entire conversation. Techniques for managing context: summarize older messages when approaching the limit, use vector search (RAG) to inject only relevant information rather than dumping entire documents, and use the model's built-in prompt caching for content that repeats across messages. Some models also support context window extension via techniques like RoPE scaling, but this usually comes with a quality trade-off.

Verdict

The window covers every token in the conversation: system prompt, user messages, assistant responses, and tool calls. A 100K token document in a 128K window leaves only 28K for the answer and follow-ups, so the limit is shared rather than per turn. RAG lets you inject just the relevant slice instead of dumping whole files, and built-in prompt caching shrinks the cost for content that repeats across turns. RoPE scaling extends the window but trades quality for length.