Skip to content
BENIBIX Learn

Masked Language Model vs Next-Token Prediction

People think every language model learns by guessing the next word. Many do, but not all. A masked model hides words in the middle and guesses them from both sides, which makes it good at reading. A next-word model sees only the words before the gap, which is what lets it write. BERT learned the first way, and chat models learn the second.

AI🔤 NLP & Language

Masked Language Model

A model trained by hiding words in text and guessing them back. BERT learned this way.

A masked language model learns by guessing hidden words. During training, some words in a sentence are covered up with a mask. The model sees the rest of the sentence, on both sides, and must guess the covered words. BERT trained this way, hiding 15% of the tokens in each piece of text. It needs no labels, because the text itself holds the answers. This makes a model that reads well, not one that writes.

Example
The model sees 'The cat sat on the [MASK].' It gives 'mat' a high chance and 'floor' a lower one.
Why it matters
It lets a model learn from huge amounts of plain text with no human labels. That made strong reading models cheap to build.
Full entry
AI💬 LLMs

Next-Token Prediction

Guessing the next token from the text so far. Repeat it and the model writes.

Next-token prediction is the one job a large language model is trained to do. It looks at the text so far. Then it gives a chance for every token that could come next. To write, the model picks one token, adds it to the text, and guesses again. A long answer is just this loop run hundreds of times. In training, the model is scored on how well it guessed each real next token.

Example
Emma types 'The capital of Italy is'. The model gives ' Rome' a chance of 0.97, picks it, then predicts what comes after ' Rome'.
Why it matters
It explains how these models write and why they can drift. Each token depends only on the ones before it.
Full entry

In one line

Masked Language Model
A model trained by hiding words in text and guessing them back. BERT learned this way.
Next-Token Prediction
Guessing the next token from the text so far. Repeat it and the model writes.