The context window: an LLM's working memory
A language model has no memory between messages. Every time you hit send, the app gives it the whole conversation again: instructions, every earlier message, tool results, all of it. The maximum amount of text it can take in at once is its context window, measured in tokens.
Models don't remember. Apps resend.
It feels like a chatbot remembers what you said five messages ago. It doesn't. On your sixth message, the app sends something like this:
[system] You are a helpful assistant… + tool definitions
[user] message 1
[assistant] reply 1
…
[user] message 6 ← the new one
The model reads all of it from scratch, then writes reply 6. Its "memory" is exactly what fits in this package. Anything not in the context window does not exist for the model.
How big is it?
It depends on the model and how it's run:
- The small models in the playground run with a 4,096-token window. That's about 3,000 words, a handful of back-and-forth messages.
- Large hosted models commonly offer 128,000 tokens or more, and some reach a million.
The window holds the input and the reply being written. If 4,000 of 4,096 tokens are used by the conversation, the model only has room for about 96 tokens of answer.
What fills the window
- The system prompt, sent with every request.
- Tool definitions. Every enabled tool's description and JSON Schema. Enable five tools and you've used a few hundred tokens before saying hello.
- The conversation history, both your messages and the model's replies.
- Tool calls and results. A tool call and its result stay in the history. A Wikipedia search result can be bigger than your whole question.
- Thinking. Reasoning models write a hidden chain of thought, and those tokens count while they're being generated.
What happens when it's full
Something has to give. Depending on the app:
- The request fails with an error that the input is too long.
- The app drops the oldest messages. The model "forgets" how the conversation started, and that's often when chatbots lose track of instructions or facts you gave them early on.
- The app summarises older messages into a shorter note, losing detail.
Even before it's full, very long contexts can hurt quality. Models tend to pay most attention to the beginning and the end, and can miss details buried in the middle.
The next lesson, managing the context window, compares these strategies side by side.
Longer context = slower and more expensive
Every token in the window has to be processed before the first word of the answer appears. That's the prefill phase. A longer conversation means a longer wait, more memory for the model's cache and, on paid APIs, a higher bill on every single message.
Practical tips
- Start a new chat when you change topic. Old messages cost tokens and can confuse the model.
- Keep the system prompt short, and only enable the tools you need.
- Put important instructions near the end of a long prompt, or repeat them.
- For long documents, send only the relevant parts. That's the idea behind "retrieval" (RAG) systems.