Lesson 2 of 8 · 4 min read

Temperature, top-p and top-k: how an LLM picks the next token

A model doesn't output a word. For every possible next token it outputs a probability, and then something has to pick one. That picking step is called sampling, and temperature, top-p and top-k are the knobs that control it.

The model gives you odds, not an answer

Say the text so far is My favourite fruit is. The model scores every token in its vocabulary (often 100,000+) and turns the scores into probabilities that add up to 100%. The top few might look like this:

 mango   40%
 apple   25%
 banana  15%
 cherry  10%
 durian   6%
 kiwi     4%
 ...      (everything else, tiny)

The app then rolls a weighted die: mango comes up 40% of the time, kiwi 4%. The chosen token is appended to the text, and the whole thing repeats for the next token. That's why the same question can get a different answer each time.

Temperature: sharpen or flatten the odds

Temperature reshapes the probabilities before the roll. Below 1 it makes likely tokens even likelier. Above 1 it spreads the chances toward the unlikely ones.

Next-token probabilities for My favourite fruit is at three temperatures
TokenT = 0.5T = 1T = 1.5
mango61%40%32%
apple24%25%23%
banana9%15%17%
cherry4%10%13%
durian1%6%9%
kiwi1%4%7%

Under the hood each raw score is divided by the temperature before being turned into probabilities, which is why small temperatures exaggerate the gaps between tokens.

Top-k: only the k best tokens

Top-k throws away everything except the k likeliest tokens, then rolls among those. With top_k = 2 only mango and apple survive, rescaled to 62% and 38%. Even at high temperature, durian can no longer appear.

The weakness: k is fixed. When the model is sure (one token at 95%) k = 40 still lets 39 bad options in. When it's unsure (many good options) a small k cuts good ones out.

Top-p: the smallest set that covers p

Top-p (also called nucleus sampling) adapts to that. It keeps adding tokens from most to least likely until their probabilities add up to p, then drops the rest. With top_p = 0.8: mango (40) + apple (25) + banana (15) = 80%, so those three stay. If the model were 95% sure of one token, top-p 0.8 would keep only that one.

Typical order: temperature reshapes the odds, then top-k and top-p cut the tail, then the die is rolled. Most APIs suggest changing temperature or top-p, not both, because they push in similar directions.

Seed and max tokens

Not every server accepts every knob

Temperature, top-p, seed and max tokens are part of the OpenAI chat API, so almost every server understands them. Top-k is not. Ollama, vLLM and llama.cpp support it in their own APIs, but OpenAI and Groq reject a request that contains it, and in-browser WebLLM ignores it. If a request fails right after you set top-k, that's why.