TechCompare LogoTechCompare

What is a context window? How LLMs remember your conversation

The context window is the LLM's working memory. Respect its limits. For long documents, use RAG. For long conversations, summarize. For structured workflows, reset the context between independent tasks.

An LLM's context window is the token budget available to a request. It can include system or developer instructions, user messages, assistant responses, and tool calls. When a conversation reaches that limit, what happens depends on the API or app: it may reject the request, truncate older messages if configured, or manage history another way.

By TechCompare · Updated

Knowledge area
Fundamentals
How tokens and tokenization work
Topic focus
Context window explained
context-window
Try the live Token Counter. Paste your prompt and see how many tokens it costs across models.
Edit in the full tool

How this is calculated

The context window includes both input and output tokens. If a model has a 128K context window and you send a 100K token document, you only have 28K tokens left for the model's response and any follow-up messages. The window is shared across the entire conversation. Techniques for managing context: summarize older messages when approaching the limit, use vector search (RAG) to inject only relevant information rather than dumping entire documents, and use the model's built-in prompt caching for content that repeats across messages. Some models also support context window extension via techniques like RoPE scaling, but this usually comes with a quality trade-off.

Verdict

The window covers every token in the conversation: system prompt, user messages, assistant responses, and tool calls. A 100K token document in a 128K window leaves only 28K for the answer and follow-ups, so the limit is shared rather than per turn. RAG lets you inject just the relevant slice instead of dumping whole files, and built-in prompt caching shrinks the cost for content that repeats across turns. RoPE scaling extends the window but trades quality for length.

More Tokens scenarios

What are tokens
A token is the atomic unit of text that a language model processes.
View details ➜
OpenAI vs Llama tokenizer
A sentence that costs 50 tokens on GPT-5 might cost 55 tokens on Llama 4 or 48 on Claude.
View details ➜
Token limits by model
A model context limit is a token budget, but providers may publish separate input and output limits.
View details ➜

Related guides

LLM token pricing comparison
Pricing on a per-token basis makes context window size a budget decision.
View guide ➜
LLM VRAM for 128K context
How the Llama 3.1 8B's 128K-context variant pushes KV cache VRAM limits.
View guide ➜

Frequently asked questions

What counts toward an LLM's context window?
Everything in the conversation, both directions: the system prompt, every user message, every assistant response, and tool call payloads. A 128K window holding a 100K-token document has only 28K left for the reply and follow-ups. The window is shared across the whole session, not allocated per turn.
Why does the model forget things I said earlier in a long chat?
The window filled and the oldest tokens dropped out. Once earlier messages fall outside the context window, the model has no memory of them - it isn't being forgetful so much as the input physically no longer contains that text. Summarizing the session and restarting clean, or injecting only relevant slices via RAG, keeps important details alive.
Is RAG better than a huge context window for long documents?
Usually. RAG retrieves the few chunks that match the question and injects only those, which keeps cost down and dodges the lost-in-the-middle attention dip that very long contexts suffer. A huge window is simpler to wire up, but for repeated queries over a large corpus, retrieval is cheaper per question and often more accurate.

Related tools

LLM API Pricing Calculator

Compare API costs across major models (OpenAI, Anthropic, Google) with prompt caching.

Use tool ➜

LLM VRAM Calculator

Calculate the VRAM needed to run or fine-tune any LLM at any quantization.

Use tool ➜

JSON Formatter

Validate, format, and minify JSON data with readable output and error detection.

Use tool ➜