Skip to content
BENIBIX Learn

Dropout

Also known as Dropout Regularization

Random neurons switch off in training. The rest learn to cope.

Think of it like

Think of it like a team where some players sit out each game. Then all of them learn every job.

Example

Jack adds dropout of 0.3 to his network. At each step, 3 in 10 neurons sit out. His validation score goes up by 5 points.

Why it matters

It is cheap, simple and works on many networks. It was a big step in making deep networks practical.

Definition

Dropout switches off random neurons during training so a network cannot memorise. At each step, some neurons, often 10 to 50 percent, are turned off at random. The rest have to do the job without them. This stops the network from leaning too hard on any one neuron. When the model is used for real, all neurons are switched back on.
On the right, the crossed-out neurons sit out for one step. Next step, a different set sits out.

Common mistake

Leaving dropout on when the model is used for real. It should only be on during training.

Key takeaways

  1. 1
    Switch off at random. Some neurons sit out.
  2. 2
    Spread the work. No neuron is leaned on too hard.
  3. 3
    Training only. All neurons back on in use.

Learn it properly