# Massive Multitask Language Understanding (MMLU)

> A test of about 16,000 questions over 57 subjects. Each has four choices.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** MMLU, MMLU Benchmark, MMLU-Pro

## Think of it like

It is like a big quiz on every school subject. Each question has four answers to pick from.

## Definition

MMLU is a benchmark of about 16,000 multiple-choice questions across 57 subjects, from math and law to history and medicine. Each question has four answers, and the score is the share the model gets right. It came out in 2020 and became one of the most quoted scores for language models. Top models now score around 90%, so it no longer tells them apart well. Harder versions, such as MMLU-Pro, were made for that reason.

## Example

A model's release table lists MMLU at 88%. Emma reads that as strong general knowledge, not as proof it can do her legal task.

## Why it matters

For years it was the main way to compare what models know. Many old scores you read still use it.

## Common mistake

Treating small gaps near the top as real. A one-point gap on a test this close to full marks is mostly noise.

## Three principles

- **57 subjects.** About 16,000 questions.
- **Four choices.** Share right is the score.
- **Near full marks.** Harder versions now.

## Related terms

- [AI Benchmark](https://learn.benibix.com/term/ai/ai-benchmark)
- [Benchmark Contamination](https://learn.benibix.com/term/ai/benchmark-contamination)
- [SWE-bench](https://learn.benibix.com/term/ai/swe-bench)
- [Large Language Model (LLM)](https://learn.benibix.com/term/ai/large-language-model)
- [Benchmark Saturation](https://learn.benibix.com/term/ai/benchmark-saturation)
- [Graduate-Level Google-Proof Q&A (GPQA)](https://learn.benibix.com/term/ai/gpqa)

![One sample question with four choices, labeled A to D. Around it, subject tags such as math, law, history and medicine show the range of the test.](https://learn.benibix.com/diagrams/ev-mmlu.svg)

---

Source: https://learn.benibix.com/term/ai/mmlu
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
