Lesson 3 of 8 · 3 min read

The context window: an LLM's working memory

A language model has no memory between messages. Every time you hit send, the app gives it the whole conversation again: instructions, every earlier message, tool results, all of it. The maximum amount of text it can take in at once is its context window, measured in tokens.

Models don't remember. Apps resend.

It feels like a chatbot remembers what you said five messages ago. It doesn't. On your sixth message, the app sends something like this:

[system]     You are a helpful assistant…  + tool definitions
[user]       message 1
[assistant]  reply 1
…
[user]       message 6   ← the new one

The model reads all of it from scratch, then writes reply 6. Its "memory" is exactly what fits in this package. Anything not in the context window does not exist for the model.

How big is it?

It depends on the model and how it's run:

The window holds the input and the reply being written. If 4,000 of 4,096 tokens are used by the conversation, the model only has room for about 96 tokens of answer.

What fills the window

What happens when it's full

Something has to give. Depending on the app:

Even before it's full, very long contexts can hurt quality. Models tend to pay most attention to the beginning and the end, and can miss details buried in the middle.

The next lesson, managing the context window, compares these strategies side by side.

Longer context = slower and more expensive

Every token in the window has to be processed before the first word of the answer appears. That's the prefill phase. A longer conversation means a longer wait, more memory for the model's cache and, on paid APIs, a higher bill on every single message.

Practical tips