Skip to content
BENIBIX Learn

Validation Set

Also known as Dev Set, Development Set, Holdout Set

Data kept out of training. You use it to pick the best settings.

Think of it like

Think of it like a practice quiz. You use it to see what to study more.

Example

Jack trains five versions of his model with different settings. He checks all five on the validation set and keeps the one that scores best.

Why it matters

It lets you make the model better. The test set stays clean for one fair final check.

Definition

The validation set is a part of the data kept out of training and used to tune the model. After each try, you check the model on it. Then you change a setting and try again. This lets you pick the best settings without touching the test set. Because you look at it many times, its score gets a little too hopeful.
The validation set is the middle part. You check on it often while you tune.

Common mistake

Reporting the validation score as the final score. You chose the model by it, so it flatters the model a little.

Key takeaways

  1. 1
    Kept out of training. The model does not learn from it.
  2. 2
    Used for tuning. Compare settings on it.
  3. 3
    Checked many times. So its score runs a bit high.

Compare

Learn it properly