# Benchmark Contamination

> When a model saw the test questions while it learned. Its score is then too high.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** Data Contamination, Test Set Contamination, Benchmark Leakage

## Think of it like

It is like a kid who saw the test answers the night before. A perfect score proves nothing.

## Definition

Benchmark contamination is when a model saw a test's questions or answers while it was trained. Big models learn from huge amounts of web text, and public tests are often posted online. So this can happen by accident. The model may then just remember the answers, and its score looks higher than its real skill. Model makers try to take test questions out of the data they train on. Some benchmarks keep their questions secret, or add new ones over time.

## Example

A model scores 85% on an old coding benchmark but only 50% on new tasks of the same kind. Lucy suspects the old questions were in its training data.

## Why it matters

It quietly breaks the main way models are compared. A score like this rewards memory, not skill.

## Common mistake

Trusting a score on a benchmark that has been public for years. Prefer fresh or hidden tests when the result matters.

## Three principles

- **Saw the test.** In training data.
- **Memory, not skill.** Score too high.
- **Fresh or secret tests.** The fix.

## Related terms

- [AI Benchmark](https://learn.benibix.com/term/ai/ai-benchmark)
- [Data Leakage](https://learn.benibix.com/term/ai/data-leakage)
- [Training Data](https://learn.benibix.com/term/ai/training-data)
- [Eval Set](https://learn.benibix.com/term/ai/eval-set)
- [Massive Multitask Language Understanding (MMLU)](https://learn.benibix.com/term/ai/mmlu)
- [Benchmark Saturation](https://learn.benibix.com/term/ai/benchmark-saturation)

## Told apart from

- [Benchmark Saturation vs Benchmark Contamination](https://learn.benibix.com/compare/benchmark-saturation-vs-benchmark-contamination): People think a test every model aces must have leaked into training. Sometimes it has, but not always. Saturation means the test got too easy for today's models. Contamination means the answers were in the training data. Both give high scores. Only one is a kind of cheat, and each needs its own fix.

![Test questions posted on the web end up in the training data. The model then scores 85% on the old test but only 50% on fresh questions of the same kind.](https://learn.benibix.com/diagrams/ev-benchmark-contamination.svg)

---

Source: https://learn.benibix.com/term/ai/benchmark-contamination
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
