📏Benchmark Contamination
Also known as Data Contamination, Test Set Contamination, Benchmark Leakage
When a model saw the test questions while it learned. Its score is then too high.
Think of it like
It is like a kid who saw the test answers the night before. A perfect score proves nothing.
Example
A model scores 85% on an old coding benchmark but only 50% on new tasks of the same kind. Lucy suspects the old questions were in its training data.
Why it matters
It quietly breaks the main way models are compared. A score like this rewards memory, not skill.
Definition
Common mistake
Key takeaways
- 1Saw the test. In training data.
- 2Memory, not skill. Score too high.
- 3Fresh or secret tests. The fix.