Lesson 7 of 8 · 3 min read

Prefill vs decode: the two speeds of an LLM

Every LLM request has two phases. First the model reads your whole prompt: that's prefill. Then it writes the answer one token at a time: that's decode. They run at very different speeds, and knowing why explains most of what "fast" and "slow" mean for language models.

Prefill: reading the prompt

All the prompt's tokens (system prompt, tools, conversation history) are known up front, so the model can process them in parallel. GPUs are excellent at this. Prefill speeds are often hundreds to thousands of tokens per second.

Prefill also builds the KV cache, the model's internal notes about every token so far, so it doesn't have to re-read the prompt for each new word it writes.

The time until the first word appears is called time to first token (TTFT). It grows with prompt length, which is why a long context makes every reply start later.

Decode: writing the answer

Writing can't be parallel. Token 2 depends on token 1, token 3 on token 2, and so on. For every new token, the hardware has to stream essentially all of the model's weights through the processor.

So decode speed is usually limited by memory bandwidth, not raw computing power. A handy estimate:

decode tokens/sec  ≈  memory bandwidth  ÷  model size in memory

e.g. 1B-parameter model, 4-bit  ≈ 0.6–0.9 GB
     laptop GPU at ~60 GB/s     →  very roughly 60–100 tok/s at best

Real numbers are lower because of overheads, but the shape holds. Smaller and more compressed (quantized) models decode faster, and faster memory means faster decode.

Putting it together

total time  ≈  time to first token  +  output tokens ÷ decode speed

prompt 1,000 tokens at 500 tok/s  →  2.0 s before the first word
answer   200 tokens at  25 tok/s  →  8.0 s of typing
                                     ≈ 10 s total

Notice that the answer dominates. Long outputs are expensive in time. That's also why reasoning models feel slow: their hidden "thinking" is decode, token by token, before the visible answer begins.

Why the second step of an agent is often faster

In a tool-calling loop, the model is called again with the same conversation plus a short tool result. Many runtimes (including WebLLM and Ollama) can reuse the KV cache for the part of the prompt they've already seen, so the second prefill only processes the new tokens.

What makes a model faster or slower