Skip to content
BENIBIX Learn

Hyperparameter Tuning

Also known as Hyperparameter Optimization, Hyperparameter Search, Grid Search

Train a model many times, each with new settings. Keep the best.

Think of it like

It is like finding the best oven heat for a cake. You bake a few and taste each one.

Example

Jack tries three tree depths and three learning rates. That makes nine models. The one with depth 6 and rate 0.1 scores best on his validation set.

Why it matters

The same model can go from poor to good with better settings. Tuning is often the cheapest way to gain that.

Definition

Hyperparameter tuning is the search for the best settings to train a model with. Settings like the learning rate or the depth of a tree are chosen before training starts. You train the model many times, each with different settings. Then you score each one on the validation set and keep the best. A grid search tries every mix on a list. A random search tries mixes picked by chance, and often finds a good one sooner.

Example code

python
from sklearn.model_selection import GridSearchCV
search = GridSearchCV(model, {"max_depth": [3, 6, 9]}, cv=5)
search.fit(X, y)
print(search.best_params_)
# {'max_depth': 6}
Three depths tried, each scored five times. The best one is printed at the end.
Each square is one trained model. The search keeps the square that scores best.

Common mistake

Scoring the settings on the test set. Then the test set no longer gives an honest final score.

Key takeaways

  1. 1
    Try many. Each setting mix is one training run.
  2. 2
    Score fairly. Use the validation set, not the test set.
  3. 3
    Search smart. Random search often beats a full grid.

Compare

Learn it properly