Skip to content
BENIBIX Learn

Masked Language Model

AI🔤 NLP & LanguageEncoder and Decoder Models

Also known as Masked Language Modeling, Masked LM

A model trained by hiding words in text and guessing them back. BERT learned this way.

Think of it like

It is like a quiz with blanks. Some words are hidden, and you guess them.

Example

The model sees 'The cat sat on the [MASK].' It gives 'mat' a high chance and 'floor' a lower one.

Why it matters

It lets a model learn from huge amounts of plain text with no human labels. That made strong reading models cheap to build.

Definition

A masked language model learns by guessing hidden words. During training, some words in a sentence are covered up with a mask. The model sees the rest of the sentence, on both sides, and must guess the covered words. BERT trained this way, hiding 15% of the tokens in each piece of text. It needs no labels, because the text itself holds the answers. This makes a model that reads well, not one that writes.
One word is hidden, and the model scores its guesses using the words around it.

Common mistake

Mixing it up with next-word prediction. Chat models guess the next word from the left only, while a masked model uses both sides.

Key takeaways

  1. 1
    Hide and guess. Cover words, then predict them.
  2. 2
    Both sides. Uses the words before and after.
  3. 3
    No labels. The text is its own answer key.

Compare