Skip to content
BENIBIX Learn

Validation Set vs Test Set

People often think these are two names for the same held-out data. Both are kept out of training, but they have different jobs. The validation set is checked again and again while you tune. The test set is opened once, at the end. Tuning on the test set makes your final score a lie.

AI📈 Machine Learning

Validation Set

Data kept out of training. You use it to pick the best settings.

The validation set is a part of the data kept out of training and used to tune the model. After each try, you check the model on it. Then you change a setting and try again. This lets you pick the best settings without touching the test set. Because you look at it many times, its score gets a little too hopeful.

Example
Jack trains five versions of his model with different settings. He checks all five on the validation set and keeps the one that scores best.
Why it matters
It lets you make the model better. The test set stays clean for one fair final check.
Full entry
AI📈 Machine Learning

Test Set

Data kept locked away. You use it once, at the end, for a fair final score.

The test set is data kept locked away until the model is finished. It is used once, for the final score. Because the model never saw it, the score shows how well it will do on new data. If you peek at it while you build the model, it stops being a fair test. Many teams keep it hidden even from the people training the model.

Example
Lucy's team finishes its spam model. Only then do they run it on 2,000 emails nobody has touched. It catches 96 percent of the spam.
Why it matters
It is your best guess of how the model will do for real users. A score from anywhere else is too hopeful.
Full entry

In one line

Validation Set
Data kept out of training. You use it to pick the best settings.
Test Set
Data kept locked away. You use it once, at the end, for a fair final score.