# Regularization and Dropout

> Ways to stop a model from learning its data by heart. A cost on big weights, neurons switched off at random, and stopping at the right epoch.

**Course:** [Machine Learning](https://learn.benibix.com/learn/ai-machine-learning)  
**Time:** 12 min

**After this lesson you can:**

- Explain what regularization trades away
- Describe how dropout works
- Use early stopping with a validation set

## How to stop a model from memorising

Regularization is any way to stop a model from fitting its training data too closely. The aim is a better score on new data, even if the training score drops a little. The three most used are a cost on big weights, dropout and early stopping. Most networks use at least two of them.

The cost adds a small amount to the loss whenever weights grow large. To keep the loss low, training now has to keep the weights small as well as fit the data. Small weights make smoother, simpler models, which follow the pattern rather than the noise.

Dropout switches off random neurons during training. At each step, a share of them sit out, often half in the hidden layers. The rest must do the job without them. So no neuron can lean too hard on any other, and the network learns several ways to reach the answer.

Dropout is only for training. When the model is put to use, every neuron is switched back on. Libraries do this when you put the model in its use mode. Forgetting that switch is a common bug, and it makes answers change from one run to the next.

Early stopping watches the loss on the validation set after each epoch. At first it falls along with the training loss. Then it flattens or starts to rise, which means the network has begun to memorise. Training stops there, and the best version so far is kept.

A patience setting stops it from quitting too soon. With a patience of 5, training goes on for 5 more epochs after the best one, in case the rise was just a bump. If nothing beats the best, it stops and goes back to it.

People often add every kind at full strength, then wonder why the model underfits. Each one is a dial, and too much is as bad as too little. Start with one. Watch the gap between the training and validation scores, and turn it up only while the gap shrinks.

![A chart of loss against epochs. The training loss keeps falling. The validation loss falls, reaches its lowest point and turns up. A line marks stop here at that lowest point, and to the right of it the model is learning noise.](https://learn.benibix.com/diagrams/nn-overfit-curve.svg)

### Example: Jack stops early

Jack plans 100 epochs with a patience of 5. The validation loss is lowest at epoch 23, then rises for the next 5.

Training stops at epoch 28 and keeps the version from epoch 23. He adds dropout of 0.3, and his validation score goes up by 4 points.

**Words in this lesson:** [Regularization](https://learn.benibix.com/term/ai/regularization), [Dropout](https://learn.benibix.com/term/ai/dropout), [Early Stopping](https://learn.benibix.com/term/ai/early-stopping)

**Question:** What does dropout do during training?

**Answer:** Switches off random neurons at each step. A random share of neurons sits out at each step, so the network cannot lean on any one of them. All of them come back on when the model is used.

---

Source: https://learn.benibix.com/learn/ai-machine-learning/regularization-and-dropout
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
