Skip to content
BENIBIX Learn

Perplexity (Metric)

AI📏 Evaluation & BenchmarksText Generation Metrics

Also known as Language Model Perplexity, Perplexity Score

A score for how well a language model predicts text. Lower means less surprised.

Think of it like

It is like how surprised you are by each next word in a story. Less surprise means you know the story well.

Example

A team trains two small models on news stories. On new stories, one has a perplexity of 18 and the other 25, so the first predicts the text better.

Why it matters

It is cheap to work out and needs no human grader. So it is watched all through training to see if a model is learning.

Formula

Perplexity = e to the power of (the average of -log(chance of each real next token))

Definition

Perplexity measures how well a language model predicts text it has not seen. At each step, the model gives a chance to the token that really comes next. Perplexity turns all those chances into one number. A perplexity of 10 means the model is as unsure as if it were picking from 10 equally likely words. Lower is better. It checks how well the model knows the language, not whether its answers are true.
Tall bars mean the model saw the word coming. Short bars push perplexity up.

Common mistake

Comparing the score of two models that cut text in different ways. They count different pieces, so the numbers do not line up.

Key takeaways

  1. 1
    Chance of the real token. How sure it was of what came.
  2. 2
    Lower is better. Less surprise, better fit.
  3. 3
    Not truth. Fluent is not the same as right.

Compare