Validation Set vs Test Set
People often think these are two names for the same held-out data. Both are kept out of training, but they have different jobs. The validation set is checked again and again while you tune. The test set is opened once, at the end. Tuning on the test set makes your final score a lie.
Validation Set
Data kept out of training. You use it to pick the best settings.
The validation set is a part of the data kept out of training and used to tune the model. After each try, you check the model on it. Then you change a setting and try again. This lets you pick the best settings without touching the test set. Because you look at it many times, its score gets a little too hopeful.
Test Set
Data kept locked away. You use it once, at the end, for a fair final score.
The test set is data kept locked away until the model is finished. It is used once, for the final score. Because the model never saw it, the score shows how well it will do on new data. If you peek at it while you build the model, it stops being a fair test. Many teams keep it hidden even from the people training the model.
In one line
- Validation Set
- Data kept out of training. You use it to pick the best settings.
- Test Set
- Data kept locked away. You use it once, at the end, for a fair final score.