Skip to content
BENIBIX Learn

Benchmark Saturation vs Benchmark Contamination

People think a test every model aces must have leaked into training. Sometimes it has, but not always. Saturation means the test got too easy for today's models. Contamination means the answers were in the training data. Both give high scores. Only one is a kind of cheat, and each needs its own fix.

AI📏 Evaluation & Benchmarks

Benchmark Saturation

Top models all score near full marks. So the test can no longer tell them apart.

Benchmark saturation is when top models score so close to full marks that a test stops telling them apart. A gap of one or two points near the top is mostly noise. High scores can also mean that answers leaked into the training data. When a test saturates, people build a harder one. MMLU went this way, with top models over 90%, and harder tests such as Humanity's Last Exam were made to replace it.

Example
Three models score 91%, 92% and 92.5% on an old quiz. Jack cannot tell which is best, so he looks at a newer, harder test.
Why it matters
A saturated test makes every new model look the same. Teams need to know which tests still have room to show a difference.
Full entry
AI📏 Evaluation & Benchmarks

Benchmark Contamination

When a model saw the test questions while it learned. Its score is then too high.

Benchmark contamination is when a model saw a test's questions or answers while it was trained. Big models learn from huge amounts of web text, and public tests are often posted online. So this can happen by accident. The model may then just remember the answers, and its score looks higher than its real skill. Model makers try to take test questions out of the data they train on. Some benchmarks keep their questions secret, or add new ones over time.

Example
A model scores 85% on an old coding benchmark but only 50% on new tasks of the same kind. Lucy suspects the old questions were in its training data.
Why it matters
It quietly breaks the main way models are compared. A score like this rewards memory, not skill.
Full entry

In one line

Benchmark Saturation
Top models all score near full marks. So the test can no longer tell them apart.
Benchmark Contamination
When a model saw the test questions while it learned. Its score is then too high.