📏Perplexity (Metric)
Also known as Language Model Perplexity, Perplexity Score
A score for how well a language model predicts text. Lower means less surprised.
Think of it like
It is like how surprised you are by each next word in a story. Less surprise means you know the story well.
Example
A team trains two small models on news stories. On new stories, one has a perplexity of 18 and the other 25, so the first predicts the text better.
Why it matters
It is cheap to work out and needs no human grader. So it is watched all through training to see if a model is learning.
Formula
Perplexity = e to the power of (the average of -log(chance of each real next token))Definition
Common mistake
Key takeaways
- 1Chance of the real token. How sure it was of what came.
- 2Lower is better. Less surprise, better fit.
- 3Not truth. Fluent is not the same as right.