Masked Language Model vs Next-Token Prediction
People think every language model learns by guessing the next word. Many do, but not all. A masked model hides words in the middle and guesses them from both sides, which makes it good at reading. A next-word model sees only the words before the gap, which is what lets it write. BERT learned the first way, and chat models learn the second.
Masked Language Model
A model trained by hiding words in text and guessing them back. BERT learned this way.
A masked language model learns by guessing hidden words. During training, some words in a sentence are covered up with a mask. The model sees the rest of the sentence, on both sides, and must guess the covered words. BERT trained this way, hiding 15% of the tokens in each piece of text. It needs no labels, because the text itself holds the answers. This makes a model that reads well, not one that writes.
Next-Token Prediction
Guessing the next token from the text so far. Repeat it and the model writes.
Next-token prediction is the one job a large language model is trained to do. It looks at the text so far. Then it gives a chance for every token that could come next. To write, the model picks one token, adds it to the text, and guesses again. A long answer is just this loop run hundreds of times. In training, the model is scored on how well it guessed each real next token.
In one line
- Masked Language Model
- A model trained by hiding words in text and guessing them back. BERT learned this way.
- Next-Token Prediction
- Guessing the next token from the text so far. Repeat it and the model writes.