Skip to content
BENIBIX Learn

Overfitting vs Data Leakage

People think a leak and overfitting are the same problem, since both make a model look better than it is. Overfitting is a model learning its training data too closely. It shows as a gap between the training score and the held-back score. A data leak is when facts from the test slip into training. It hides the gap, because the held-back score is high too. More data or a simpler model fixes overfitting. Only a cleaner split fixes a leak.

AI📈 Machine Learning

Overfitting

The model learns its old data by heart. Then it fails on new data.

Overfitting is when a model learns its training data too closely, noise and all. It scores very well on the data it trained on. But on new data, it does much worse. It has memorised the examples, not the real pattern. More data, a simpler model or stopping training earlier can all help.

Example
Jack's model scores 99 percent on its training photos but only 70 percent on new ones. The gap shows it has overfit.
Why it matters
It is the most common way a model fails in real use. A great training score means little until new data agrees.
Full entry
AI📈 Machine Learning

Data Leakage

When the answers sneak into training. Test scores look great and real use fails.

Data leakage is when facts the model should not have slip into its training. It might be test examples hiding in the training set. It might be a feature that gives away the answer. The model then scores very well in testing. Later, with real data, it does much worse.

Example
Jack's model predicts which patients will get sick. One feature is the date the patient got medicine. That date gives the answer away, so the model looks perfect and fails in real use.
Why it matters
It is one of the most common reasons a model that looked great fails after launch. Nothing warns you, because the scores look fine.
Full entry

In one line

Overfitting
The model learns its old data by heart. Then it fails on new data.
Data Leakage
When the answers sneak into training. Test scores look great and real use fails.