Prefill vs decode: the two speeds of an LLM
Every LLM request has two phases. First the model reads your whole prompt: that's prefill. Then it writes the answer one token at a time: that's decode. They run at very different speeds, and knowing why explains most of what "fast" and "slow" mean for language models.
Prefill: reading the prompt
All the prompt's tokens (system prompt, tools, conversation history) are known up front, so the model can process them in parallel. GPUs are excellent at this. Prefill speeds are often hundreds to thousands of tokens per second.
Prefill also builds the KV cache, the model's internal notes about every token so far, so it doesn't have to re-read the prompt for each new word it writes.
The time until the first word appears is called time to first token (TTFT). It grows with prompt length, which is why a long context makes every reply start later.
Decode: writing the answer
Writing can't be parallel. Token 2 depends on token 1, token 3 on token 2, and so on. For every new token, the hardware has to stream essentially all of the model's weights through the processor.
So decode speed is usually limited by memory bandwidth, not raw computing power. A handy estimate:
decode tokens/sec ≈ memory bandwidth ÷ model size in memory
e.g. 1B-parameter model, 4-bit ≈ 0.6–0.9 GB
laptop GPU at ~60 GB/s → very roughly 60–100 tok/s at best
Real numbers are lower because of overheads, but the shape holds. Smaller and more compressed (quantized) models decode faster, and faster memory means faster decode.
Putting it together
total time ≈ time to first token + output tokens ÷ decode speed
prompt 1,000 tokens at 500 tok/s → 2.0 s before the first word
answer 200 tokens at 25 tok/s → 8.0 s of typing
≈ 10 s total
Notice that the answer dominates. Long outputs are expensive in time. That's also why reasoning models feel slow: their hidden "thinking" is decode, token by token, before the visible answer begins.
Why the second step of an agent is often faster
In a tool-calling loop, the model is called again with the same conversation plus a short tool result. Many runtimes (including WebLLM and Ollama) can reuse the KV cache for the part of the prompt they've already seen, so the second prefill only processes the new tokens.
What makes a model faster or slower
- Model size: fewer parameters means less to read per token.
- Quantization: storing weights in 4 bits instead of 16 cuts memory traffic roughly 4×, at a small cost in quality.
- Hardware: GPU memory bandwidth matters most for decode, compute for prefill.
- Prompt length: longer prompts mean longer prefill and a bigger KV cache.
- Network and queueing: for hosted APIs, part of the "prefill" you measure is just waiting.