Skip to content
BENIBIX Learn

Gradient Descent and Backprop

Level 5 · Neural networks

How a neural network learns. A loss score, backpropagation to find each weight's share of the error, and gradient descent to fix it, batch by batch.

Course contentsLesson 14 of 18

After this lesson you can

  • Explain what a loss function measures
  • Describe one training step
  • Pick a sensible learning rate, batch size and number of epochs

14 mincovers 7 glossary entries

Worth reading first: Neural Network Basics

How a neural network learns from its mistakes

A neural network learns by repeating one step thousands of times. It makes a guess, measures how wrong it was, traces the error back to every weight, and nudges each weight to shrink it. The way of tracing the error, backpropagation, was made famous by a 1986 paper. Almost every network still trains this way.

The measure of wrongness is the loss function. It compares each guess with the right answer and gives back one number, the loss. Guess a house at 300,000 euros when it sold for 340,000, and the loss is large. Training has one goal: make that number small.

Backpropagation works out how much each weight added to the loss. It starts at the output and passes the error backwards, layer by layer. At each weight it gets a number, called the gradient, that says which way to move the weight to lower the loss, and how strongly.

Gradient descent then makes the move. Every weight takes a small step in the direction that lowers the loss. The loss is like a hilly field, and the model is a walker in fog, feeling the slope underfoot and stepping downhill. Many small steps reach a valley.

The learning rate sets the size of each step. Too large, and the walker jumps over the valley and the loss bounces about. Too small, and training crawls. The optimizer is the rule that turns gradients into steps. Adam, from 2014, adjusts the step for each weight as it goes, with a default rate of 0.001.

Training does not use all the data in each step. It splits the rows into batches, often 32 or 64 at a time, and takes one step after each batch. One full pass through all the rows is an epoch. With 1,000 rows and a batch size of 100, one epoch is ten steps.

People think more epochs always help. At first, the loss on new data falls along with the training loss. After a point it starts to rise, which means the network is memorising. More epochs past that point make the model worse, not better.

One training step, repeated for every batch. Thousands of them make one training run.

Worked example

Emma fixes her step size

Emma trains a photo model with a learning rate of 0.5. The loss jumps between 2 and 9 and never settles.

She drops the rate to 0.001 and switches to Adam. Over 20 epochs of 50 batches each, the loss falls smoothly from 2.3 to 0.4.

The words for this lesson7

Check yourself

What does backpropagation work out?

Finished this lesson?

Marking it complete also marks its 7 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.