What is a token?
When you type a message to a chatbot, the model never sees your words. Your text is first chopped into tokens: small pieces of text, each mapped to a number. The model reads numbers, predicts numbers, and those numbers are turned back into text. Almost everything about LLMs is measured in tokens: limits, prices and speed.
A token is a piece of text
A token can be a whole word, part of a word, a single character, or punctuation. Common words are usually one token. Rare or long words are split into several:
"The cat sat" → ["The", " cat", " sat"] 3 tokens
"unbelievably" → ["un", "belie", "vably"] 3 tokens (example split)
"ChatGPT" → ["Chat", "G", "PT"] 3 tokens (example split)
Notice the leading spaces: " cat" with a space is a different token from "cat". The exact split depends on the model's tokenizer. Qwen, Llama and GPT models each have their own, so the same sentence can be a different number of tokens on each.
A useful rule of thumb for English: 1 token ≈ 4 characters ≈ ¾ of a word. So 1,000 tokens is roughly 750 words.
Tokens are numbers
Each token in the tokenizer's vocabulary has an ID. Modern models have vocabularies of around 32,000 to 150,000+ tokens. The model's input is a list of these IDs:
"The cat sat" → [791, 8415, 7731] (IDs are illustrative)
Inside the model, each ID is turned into a long list of numbers (an embedding) that captures something about its meaning. Everything after that is math on those numbers.
Why not just use words or letters?
- Words would need a gigantic vocabulary and still fail on typos, names, code and new words.
- Letters would make every text extremely long, and models get slower and more expensive as input grows.
- Sub-word tokens are the compromise. Frequent text gets short codes, and anything else can still be spelled out from smaller pieces. Most tokenizers learn their pieces from data using an algorithm called byte-pair encoding (BPE).
The model writes one token at a time
Generation is a loop. The model looks at all the tokens so far and outputs a probability for every token in its vocabulary. One is picked (settings like temperature control how adventurous that pick is), added to the text, and the loop repeats until the model produces a special end token.
That's why answers stream in word by word, and why speed is measured in tokens per second. See prefill vs decode.
Special tokens you never see
Chat models also use special tokens to mark structure: where the system prompt starts, where a user message ends, whose turn it is. Qwen models use markers like <|im_start|> and <|im_end|>, and Llama 3 uses <|start_header_id|>. Your conversation is wrapped in these by a chat template before the model sees it. They take up space too.
Why tokens explain strange LLM behaviour
- Counting letters is hard. Ask "how many r's are in strawberry?" and the model sees a few tokens, not ten letters. It can't easily look inside a token.
- Arithmetic is shaky. Numbers are split into chunks that don't line up with digits, which is one reason small models need a calculator tool.
- Other languages cost more. Tokenizers are trained mostly on English, so the same sentence in Indonesian, Thai or Japanese often needs more tokens. That means more cost, slower replies and less room in the context window.
- Prices and limits are per token. APIs bill per input and output token, and every model has a maximum number of tokens it can handle at once.