Managing the context window: how chat apps make room
The model re-reads the whole conversation on every turn, and the context window has a hard limit. Long before you hit it, every extra token makes each answer slower and more expensive. So real apps compact the conversation: they decide what the model keeps seeing and what it forgets.
What you see vs what the model sees
A chat app keeps two lists. One is the conversation on your screen. The other is the list of messages it actually sends to the model. They start out identical, but compacting changes only the second one. You can still scroll up and read your first message after the model has stopped seeing it, and that gap is why chatbots "forget" things that are right there on the page.
One conversation, four strategies
Take a chat of four turns: a question answered with a Wikipedia lookup (the tool result alone is ~1,900 tokens), a turn where you give your name, a long explanation, and a short follow-up. With a ~450-token system prompt, it fills about 3,000 tokens. Here is what each strategy leaves:
| Strategy | Tokens | Keeps | Loses |
|---|---|---|---|
| Before compacting | ≈3,000 | everything | – |
| Drop oldest 4 messages | ≈990 | the last three turns | the whole Wikipedia question and its answer |
| Remove tool traffic | ≈1,070 | every question and answer | the raw tool output |
| Summarize everything 1 model call | ≈582 | a ~120-token summary | exact wording, small details |
| Summary + keep last 2 1 model call | ≈667 | summary + the last turn word for word | older details |
The system prompt and tool list are sent every time and are never compacted, which is why no strategy goes below ~450 here.
1. Drop the oldest messages
Also called a sliding window: keep the last N messages, forget the rest. It's free, instant and predictable, and it's what many apps do silently when the window fills. The cost is that facts from the start (your name, the task, an early instruction) disappear completely.
One detail matters: cut on a turn boundary. If the kept part starts with an assistant reply or a tool result, the model sees an answer to a question it never got.
2. Remove tool traffic
With function calling, a single question can produce a tool call, a large tool result, and only then an answer. Once the answer is written, the raw result is usually dead weight. Dropping the calls and results, while keeping questions and answers, often saves the most for the least loss. Coding agents do the same with old file reads and command output.
3. Summarize everything
Ask the model to rewrite the conversation as a short summary, then replace the history with that one note. This shrinks the most and keeps facts from the whole chat, but it has costs:
- One extra model call, which reads the entire conversation once more.
- Details get lost. Exact numbers, code and wording become "the user asked about X".
- Small models summarize badly. A 1–2B model may drop important facts or invent ones that were never said. Always check the summary before relying on it.
4. Summary + keep the recent messages
The usual compromise: summarize the older part and keep the last few messages word for word. The model keeps the overall story and the exact wording of what you're working on right now. This is roughly what ChatGPT-style apps and coding agents do when they "auto-compact" near the limit.
Other ways to save context
- Pin what matters. Put facts that must survive (who the user is, the task, hard rules) in the system prompt, which is never compacted.
- Retrieve instead of carrying. Store documents or old messages outside the chat and fetch only the relevant pieces when needed. That's the idea behind RAG and "memory" features.
- Drop old reasoning. Thinking models write long hidden reasoning. Apps keep it while the model is still working on your question, then remove it from the history. It costs tokens on every later call, and it can quietly carry facts you meant to compact away. This playground does the same.
- Start a new chat when the topic changes. It's the cheapest compaction there is.