💬Tokenizer
Also known as Tokeniser, Byte-Pair Encoding, BPE
The tool that cuts text into tokens. It gives each one a number.
Think of it like
Think of it like a pair of scissors. It cuts text into the bits a model knows.
Example
Jack pastes the same paragraph into two token counters. One model's tokenizer counts 210 tokens and the other counts 245, so the second costs more to run.
Why it matters
The tokenizer sets how many tokens your text costs. It also shows why models can trip on spelling and on how many letters a word has.
Definition
Common mistake
Key takeaways
- 1Splits text. Into known pieces.
- 2A fixed vocabulary. Tens of thousands of pieces.
- 3One per model. Counts differ between models.