Skip to content
BENIBIX Learn

Next-Token Prediction

AI💬 LLMsLLM Basics

Also known as Next-Word Prediction, Autoregressive Generation

Guessing the next token from the text so far. Repeat it and the model writes.

Think of it like

Think of it like a game. You add one word to a story, then pass it on.

Example

Emma types 'The capital of Italy is'. The model gives ' Rome' a chance of 0.97, picks it, then predicts what comes after ' Rome'.

Why it matters

It explains how these models write and why they can drift. Each token depends only on the ones before it.

Definition

Next-token prediction is the one job a large language model is trained to do. It looks at the text so far. Then it gives a chance for every token that could come next. To write, the model picks one token, adds it to the text, and guesses again. A long answer is just this loop run hundreds of times. In training, the model is scored on how well it guessed each real next token.
This loop is behind every answer. Read the text, guess the next token, add it, and read again.

Common mistake

Thinking the model plans the whole answer first. It commits to one token at a time and cannot go back and change one.

Key takeaways

  1. 1
    One token at a time. A chance for each option.
  2. 2
    Pick, add, repeat. That loop writes the text.
  3. 3
    The training goal. Scored on each real next token.

Compare

Learn it properly