Skip to content
BENIBIX Learn

Hyperparameter Tuning

Level 4 · Classic models

The settings you pick before training, from tree depth to k, and how to search for the best ones on the validation set. Grid search and random search.

Course contentsLesson 12 of 18

After this lesson you can

  • Tell a hyperparameter from a parameter
  • Run a grid search on the validation set
  • Say why random search often wins

11 mincovers 2 glossary entries

Worth reading first: Decision Trees and Forests

Choosing the settings a model cannot learn

A hyperparameter is a setting you choose before training starts. The model never learns it. The depth of a tree, the k in k-NN and the number of trees in a forest are all such settings. Tuning is the search for the best values. You train many versions, score each on the validation set, and keep the winner.

The two are easy to mix up. Parameters are the numbers training fills in, often millions of them. Hyperparameters are the few dials you set by hand. Change a dial, and training finds a new set of numbers.

A grid search tries every mix from a list. Three tree depths and three step sizes make nine models. It is simple and thorough, but it grows fast. Five settings with five values each make 3,125 models to train.

A random search tries mixes picked by chance instead. A 2012 study showed it often finds a good mix sooner. The reason is that only one or two settings usually matter much. A grid wastes runs on the same useful value again and again, while random picks try many more values of each.

Every model you score on the validation set leaks a little of it into your choices. Try 200 mixes and pick the best, and part of that best score is luck. That is why the test set stays locked until tuning is done. Its score is the one to report.

Tuning costs time and money. Each try is a full training run. For a small table model, 100 tries may take minutes. For a large network, one try can take days, so teams tune on a small version first, then train the big one once.

People often tune before they look at the data. A bad label or a leaked column hurts far more than a setting that is a bit off. The defaults in good libraries are sensible. Tune last, after the data and the features are right.

Nine settings make nine models. Each is scored on the same set, and one wins.

Worked example

Jack tunes his forest

Jack's random forest guesses which customers will leave. He tries 20 random mixes of tree count and tree depth.

The best scores 0.88 on the validation set, against 0.83 for the defaults. On the locked test set it scores 0.86, and that is the number he reports.

The words for this lesson2

Check yourself

Which set do you use to pick the best settings?

Finished this lesson?

Marking it complete also marks its 2 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.