Overfitting vs Underfitting
People often think a bad model has just not learned enough. That is only one way to fail. An underfit model is too simple, so it does badly on old data and new data alike. An overfit model has memorised its training data, so it shines there and fails on new data. The fixes go in opposite directions.
Overfitting
The model learns its old data by heart. Then it fails on new data.
Overfitting is when a model learns its training data too closely, noise and all. It scores very well on the data it trained on. But on new data, it does much worse. It has memorised the examples, not the real pattern. More data, a simpler model or stopping training earlier can all help.
Underfitting
The model is too simple. It does badly on old data and new data alike.
Underfitting is when a model is too simple to learn the pattern in its data. It does badly on the training data and badly on new data too. It might be a straight line used for a curved pattern. Or it might not have trained for long enough. A bigger model, better features or more training can help.
In one line
- Overfitting
- The model learns its old data by heart. Then it fails on new data.
- Underfitting
- The model is too simple. It does badly on old data and new data alike.