# Tokenizer

> The tool that cuts text into tokens. It gives each one a number.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** Tokeniser, Byte-Pair Encoding, BPE

## Think of it like

Think of it like a pair of scissors. It cuts text into the bits a model knows.

## Definition

A tokenizer splits text into tokens and gives each token its ID number. It works from a fixed list of pieces, called its vocabulary. That list often holds 50,000 to 200,000 entries. Common words get one piece of their own, and rare words are built from smaller pieces. Each model has its own, so the same text can be a different number of tokens in two models.

## Example

Jack pastes the same paragraph into two token counters. One model's tokenizer counts 210 tokens and the other counts 245, so the second costs more to run.

## Why it matters

The tokenizer sets how many tokens your text costs. It also shows why models can trip on spelling and on how many letters a word has.

## Common mistake

Thinking the model sees letters. It sees token numbers, which is why it can miss how many r's are in 'strawberry'.

## Three principles

- **Splits text.** Into known pieces.
- **A fixed vocabulary.** Tens of thousands of pieces.
- **One per model.** Counts differ between models.

## Related terms

- [Token (AI)](https://learn.benibix.com/term/ai/token)
- [Large Language Model (LLM)](https://learn.benibix.com/term/ai/large-language-model)
- [Word Embedding](https://learn.benibix.com/term/ai/word-embedding)

## Taught in

- [Tokens and Tokenizers](https://learn.benibix.com/learn/ai-llms/tokens-and-tokenizers): LLMs, 2/15

![The words unbelievable cats are split into four tokens, un, belie, vable and cats, and each token has its own number under it.](https://learn.benibix.com/diagrams/llm-tokens.svg)

---

Source: https://learn.benibix.com/term/ai/tokenizer
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
