Skip to content
BENIBIX Learn

Token (AI)

AI💬 LLMsLLM Basics

Also known as Token, LLM Token

A small piece of text, often part of a word, that a model reads.

Think of it like

Think of it like a Lego brick of text. Words are built from one or more of them.

Example

The words 'unbelievable cats' might become four tokens: 'un', 'belie', 'vable' and ' cats'. A 1,000-word English email is about 1,300 tokens.

Why it matters

Cost, speed and how much a model can read at once are all measured in tokens, not words.

Definition

A token is the small piece of text a language model reads and writes. It can be a whole word, part of a word, a space or a sign like a comma. In English, one token is about four letters, or three quarters of a word, on average. Each token has an ID number, and the model only ever sees those numbers. Prices and limits for AI tools are counted in tokens.
Two words become four tokens. Each token then becomes a number the model can work with.

Common mistake

Thinking one token is one word. Long or rare words take several, and many languages, Persian among them, need more tokens per word than English.

Key takeaways

  1. 1
    A piece of text. Word, part of a word, or sign.
  2. 2
    About four letters. In English, on average.
  3. 3
    The unit of cost. Limits and prices use it.

Learn it properly