💬Next-Token Prediction
Also known as Next-Word Prediction, Autoregressive Generation
Guessing the next token from the text so far. Repeat it and the model writes.
Think of it like
Think of it like a game. You add one word to a story, then pass it on.
Example
Emma types 'The capital of Italy is'. The model gives ' Rome' a chance of 0.97, picks it, then predicts what comes after ' Rome'.
Why it matters
It explains how these models write and why they can drift. Each token depends only on the ones before it.
Definition
Common mistake
Key takeaways
- 1One token at a time. A chance for each option.
- 2Pick, add, repeat. That loop writes the text.
- 3The training goal. Scored on each real next token.