Tokens and Tokenizers
Level 1 · What an LLM is
What a token is, and how a tokenizer cuts text into numbered pieces. Why a model writes its reply one token at a time.
Course contentsLesson 2 of 15
Level 1 · What an LLM is
Level 2 · Inside the model
Level 3 · How a model is trained
Level 4 · Choosing and running a model
Level 5 · Where LLMs go wrong
After this lesson you can
- Turn a word count into tokens
- Explain what a tokenizer does
- Describe how a model writes a reply
10 mincovers 3 glossary entries
Worth reading first: Large Language Model Basics
What a token is, and how many words it holds
An LLM does not read words. It reads tokens, which are small chunks of text. In English, one token is about four letters, or three quarters of a word. So 100 tokens is roughly 75 words. Every limit, every speed and every price for a model is counted in tokens.
A tokenizer is the tool that cuts text into tokens. It keeps a fixed list of chunks, and each chunk has an ID number. GPT-2 used a list of 50,257 chunks. Newer models use bigger lists, often over 100,000. The model only ever sees the ID numbers, never the letters.
Common words are usually one token each. A rare or long word is split into parts, so 'unbelievable' might become 'un', 'belie' and 'vable'. A space is often glued to the front of the next word. Numbers can split in odd places, which is one reason models slip on sums.
Writing works one token at a time. The model looks at all the text so far and gives every token on its list a chance of coming next. One token is picked and added to the end. Then the whole thing runs again. This loop is called next-token prediction.
That loop explains a lot. A reply of 500 tokens means the model ran 500 times. It cannot go back and fix an earlier token, because each one stays once it is added. And it does not plan the whole answer first. The answer comes out piece by piece.
Tokens are not fair across languages. Most tokenizer lists were built mainly from English text. A sentence in Persian, Hindi or Thai often needs more tokens than the same sentence in English. The same message then costs more, runs slower and fills up the model's space sooner.
People often count words when they should count tokens. A price per million tokens is not a price per million words. Code, tables and odd spellings all use more tokens than plain sentences. To know the real number, run the text through the model's own tokenizer and count.
Worked example
Lucy prices a support bot
Lucy's support bot answers about 2,000 questions a day. Each reply is around 150 words, which is about 200 tokens.
That is 400,000 tokens a day, or 12 million in a 30-day month. The price per million tokens is what she multiplies by 12, not her word count.
The words for this lesson3
Check yourself
Why does an LLM write its reply one piece at a time?
Finished this lesson?
Marking it complete also marks its 3 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.