Lesson 4 of 8 · 4 min read

Managing the context window: how chat apps make room

The model re-reads the whole conversation on every turn, and the context window has a hard limit. Long before you hit it, every extra token makes each answer slower and more expensive. So real apps compact the conversation: they decide what the model keeps seeing and what it forgets.

What you see vs what the model sees

A chat app keeps two lists. One is the conversation on your screen. The other is the list of messages it actually sends to the model. They start out identical, but compacting changes only the second one. You can still scroll up and read your first message after the model has stopped seeing it, and that gap is why chatbots "forget" things that are right there on the page.

One conversation, four strategies

Take a chat of four turns: a question answered with a Wikipedia lookup (the tool result alone is ~1,900 tokens), a turn where you give your name, a long explanation, and a short follow-up. With a ~450-token system prompt, it fills about 3,000 tokens. Here is what each strategy leaves:

Approximate tokens the model re-reads per turn after each strategy
StrategyTokensKeepsLoses
Before compacting≈3,000everything–
Drop oldest 4 messages≈990the last three turnsthe whole Wikipedia question and its answer
Remove tool traffic≈1,070every question and answerthe raw tool output
Summarize everything
1 model call
≈582a ~120-token summaryexact wording, small details
Summary + keep last 2
1 model call
≈667summary + the last turn word for wordolder details

The system prompt and tool list are sent every time and are never compacted, which is why no strategy goes below ~450 here.

1. Drop the oldest messages

Also called a sliding window: keep the last N messages, forget the rest. It's free, instant and predictable, and it's what many apps do silently when the window fills. The cost is that facts from the start (your name, the task, an early instruction) disappear completely.

One detail matters: cut on a turn boundary. If the kept part starts with an assistant reply or a tool result, the model sees an answer to a question it never got.

2. Remove tool traffic

With function calling, a single question can produce a tool call, a large tool result, and only then an answer. Once the answer is written, the raw result is usually dead weight. Dropping the calls and results, while keeping questions and answers, often saves the most for the least loss. Coding agents do the same with old file reads and command output.

3. Summarize everything

Ask the model to rewrite the conversation as a short summary, then replace the history with that one note. This shrinks the most and keeps facts from the whole chat, but it has costs:

4. Summary + keep the recent messages

The usual compromise: summarize the older part and keep the last few messages word for word. The model keeps the overall story and the exact wording of what you're working on right now. This is roughly what ChatGPT-style apps and coding agents do when they "auto-compact" near the limit.

Other ways to save context