# Hyperparameter Tuning

> The settings you pick before training, from tree depth to k, and how to search for the best ones on the validation set. Grid search and random search.

**Course:** [Machine Learning](https://learn.benibix.com/learn/ai-machine-learning)  
**Time:** 11 min

**After this lesson you can:**

- Tell a hyperparameter from a parameter
- Run a grid search on the validation set
- Say why random search often wins

## Choosing the settings a model cannot learn

A hyperparameter is a setting you choose before training starts. The model never learns it. The depth of a tree, the k in k-NN and the number of trees in a forest are all such settings. Tuning is the search for the best values. You train many versions, score each on the validation set, and keep the winner.

The two are easy to mix up. Parameters are the numbers training fills in, often millions of them. Hyperparameters are the few dials you set by hand. Change a dial, and training finds a new set of numbers.

A grid search tries every mix from a list. Three tree depths and three step sizes make nine models. It is simple and thorough, but it grows fast. Five settings with five values each make 3,125 models to train.

A random search tries mixes picked by chance instead. A 2012 study showed it often finds a good mix sooner. The reason is that only one or two settings usually matter much. A grid wastes runs on the same useful value again and again, while random picks try many more values of each.

Every model you score on the validation set leaks a little of it into your choices. Try 200 mixes and pick the best, and part of that best score is luck. That is why the test set stays locked until tuning is done. Its score is the one to report.

Tuning costs time and money. Each try is a full training run. For a small table model, 100 tries may take minutes. For a large network, one try can take days, so teams tune on a small version first, then train the big one once.

People often tune before they look at the data. A bad label or a leaked column hurts far more than a setting that is a bit off. The defaults in good libraries are sensible. Tune last, after the data and the features are right.

![A grid of nine scores, with tree depths of 3, 6 and 9 across and learning rates of 0.01, 0.1 and 0.3 down. Each cell is a score on the validation set, and the best, 0.91, sits at depth 6 and rate 0.1.](https://learn.benibix.com/diagrams/ai3-tuning-grid.svg)

### Example: Jack tunes his forest

Jack's random forest guesses which customers will leave. He tries 20 random mixes of tree count and tree depth.

The best scores 0.88 on the validation set, against 0.83 for the defaults. On the locked test set it scores 0.86, and that is the number he reports.

**Words in this lesson:** [Hyperparameter](https://learn.benibix.com/term/ai/hyperparameter), [Hyperparameter Tuning](https://learn.benibix.com/term/ai/hyperparameter-tuning)

**Question:** Which set do you use to pick the best settings?

**Answer:** The validation set. The validation set is for choices you make while building. The test set stays locked, so its score is still fair when tuning is done.

---

Source: https://learn.benibix.com/learn/ai-machine-learning/hyperparameter-tuning
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
