Skip to content
BENIBIX Learn

Tokenizer

AI💬 LLMsLLM Basics

Also known as Tokeniser, Byte-Pair Encoding, BPE

The tool that cuts text into tokens. It gives each one a number.

Think of it like

Think of it like a pair of scissors. It cuts text into the bits a model knows.

Example

Jack pastes the same paragraph into two token counters. One model's tokenizer counts 210 tokens and the other counts 245, so the second costs more to run.

Why it matters

The tokenizer sets how many tokens your text costs. It also shows why models can trip on spelling and on how many letters a word has.

Definition

A tokenizer splits text into tokens and gives each token its ID number. It works from a fixed list of pieces, called its vocabulary. That list often holds 50,000 to 200,000 entries. Common words get one piece of their own, and rare words are built from smaller pieces. Each model has its own, so the same text can be a different number of tokens in two models.
The tokenizer does both steps shown here: it cuts the text, then swaps each piece for its number.

Common mistake

Thinking the model sees letters. It sees token numbers, which is why it can miss how many r's are in 'strawberry'.

Key takeaways

  1. 1
    Splits text. Into known pieces.
  2. 2
    A fixed vocabulary. Tens of thousands of pieces.
  3. 3
    One per model. Counts differ between models.

Learn it properly