📏Massive Multitask Language Understanding (MMLU)
Also known as MMLU, MMLU Benchmark, MMLU-Pro
A test of about 16,000 questions over 57 subjects. Each has four choices.
Think of it like
It is like a big quiz on every school subject. Each question has four answers to pick from.
Example
A model's release table lists MMLU at 88%. Emma reads that as strong general knowledge, not as proof it can do her legal task.
Why it matters
For years it was the main way to compare what models know. Many old scores you read still use it.
Definition
Common mistake
Key takeaways
- 157 subjects. About 16,000 questions.
- 2Four choices. Share right is the score.
- 3Near full marks. Harder versions now.