Benchmark Saturation vs Benchmark Contamination
People think a test every model aces must have leaked into training. Sometimes it has, but not always. Saturation means the test got too easy for today's models. Contamination means the answers were in the training data. Both give high scores. Only one is a kind of cheat, and each needs its own fix.
Benchmark Saturation
Top models all score near full marks. So the test can no longer tell them apart.
Benchmark saturation is when top models score so close to full marks that a test stops telling them apart. A gap of one or two points near the top is mostly noise. High scores can also mean that answers leaked into the training data. When a test saturates, people build a harder one. MMLU went this way, with top models over 90%, and harder tests such as Humanity's Last Exam were made to replace it.
Benchmark Contamination
When a model saw the test questions while it learned. Its score is then too high.
Benchmark contamination is when a model saw a test's questions or answers while it was trained. Big models learn from huge amounts of web text, and public tests are often posted online. So this can happen by accident. The model may then just remember the answers, and its score looks higher than its real skill. Model makers try to take test questions out of the data they train on. Some benchmarks keep their questions secret, or add new ones over time.
In one line
- Benchmark Saturation
- Top models all score near full marks. So the test can no longer tell them apart.
- Benchmark Contamination
- When a model saw the test questions while it learned. Its score is then too high.