Overfitting vs Data Leakage
People think a leak and overfitting are the same problem, since both make a model look better than it is. Overfitting is a model learning its training data too closely. It shows as a gap between the training score and the held-back score. A data leak is when facts from the test slip into training. It hides the gap, because the held-back score is high too. More data or a simpler model fixes overfitting. Only a cleaner split fixes a leak.
Overfitting
The model learns its old data by heart. Then it fails on new data.
Overfitting is when a model learns its training data too closely, noise and all. It scores very well on the data it trained on. But on new data, it does much worse. It has memorised the examples, not the real pattern. More data, a simpler model or stopping training earlier can all help.
Data Leakage
When the answers sneak into training. Test scores look great and real use fails.
Data leakage is when facts the model should not have slip into its training. It might be test examples hiding in the training set. It might be a feature that gives away the answer. The model then scores very well in testing. Later, with real data, it does much worse.
In one line
- Overfitting
- The model learns its old data by heart. Then it fails on new data.
- Data Leakage
- When the answers sneak into training. Test scores look great and real use fails.