Skip to content
BENIBIX Learn

Massive Multitask Language Understanding (MMLU)

AI📏 Evaluation & BenchmarksBenchmarks and Leaderboards

Also known as MMLU, MMLU Benchmark, MMLU-Pro

A test of about 16,000 questions over 57 subjects. Each has four choices.

Think of it like

It is like a big quiz on every school subject. Each question has four answers to pick from.

Example

A model's release table lists MMLU at 88%. Emma reads that as strong general knowledge, not as proof it can do her legal task.

Why it matters

For years it was the main way to compare what models know. Many old scores you read still use it.

Definition

MMLU is a benchmark of about 16,000 multiple-choice questions across 57 subjects, from math and law to history and medicine. Each question has four answers, and the score is the share the model gets right. It came out in 2020 and became one of the most quoted scores for language models. Top models now score around 90%, so it no longer tells them apart well. Harder versions, such as MMLU-Pro, were made for that reason.
One question, four choices, and subjects from math to medicine around it.

Common mistake

Treating small gaps near the top as real. A one-point gap on a test this close to full marks is mostly noise.

Key takeaways

  1. 1
    57 subjects. About 16,000 questions.
  2. 2
    Four choices. Share right is the score.
  3. 3
    Near full marks. Harder versions now.