Skip to content
BENIBIX Learn

Backpropagation vs Gradient Descent

People often use the two names for the same thing. They are two steps that work as a pair. Backpropagation works out how much each weight added to the error. Gradient descent then uses those numbers to move every weight a small step. One finds the changes, and the other makes them.

AI📈 Machine Learning

Backpropagation

It traces the error back through the layers to see which weights to fix.

Backpropagation finds how much each weight in a network added to its error. After the network makes a guess, the loss is measured at the output. Then the error is passed backwards, layer by layer, toward the input. At each layer, it works out how much each weight should change. Gradient descent then uses those numbers to update the weights.

Example
Lucy's network says dog for a cat photo. Backpropagation traces the miss back through all 12 layers. It finds the weights that pushed hardest toward dog.
Why it matters
It makes training deep networks possible. Without a fast way to share out the blame, big networks could not learn.
Full entry
AI📈 Machine Learning

Gradient Descent

How training improves a model. Each small step nudges the weights downhill.

Gradient descent lowers a model's loss one small step at a time. At each step, it works out which way to nudge every weight so the loss goes down. Then it moves all the weights a little in that direction. It repeats this thousands or millions of times. Most training today uses a version that looks at a small batch of data per step.

Example
Jack's model starts with random weights and a loss of 8.0. After 1,000 steps of gradient descent, the loss is down to 0.3.
Why it matters
It is how almost every neural network learns. Without it, there would be no way to tune billions of weights.
Full entry

In one line

Backpropagation
It traces the error back through the layers to see which weights to fix.
Gradient Descent
How training improves a model. Each small step nudges the weights downhill.