# Gradient Descent and Backprop

> How a neural network learns. A loss score, backpropagation to find each weight's share of the error, and gradient descent to fix it, batch by batch.

**Course:** [Machine Learning](https://learn.benibix.com/learn/ai-machine-learning)  
**Time:** 14 min

**After this lesson you can:**

- Explain what a loss function measures
- Describe one training step
- Pick a sensible learning rate, batch size and number of epochs

## How a neural network learns from its mistakes

A neural network learns by repeating one step thousands of times. It makes a guess, measures how wrong it was, traces the error back to every weight, and nudges each weight to shrink it. The way of tracing the error, backpropagation, was made famous by a 1986 paper. Almost every network still trains this way.

The measure of wrongness is the loss function. It compares each guess with the right answer and gives back one number, the loss. Guess a house at 300,000 euros when it sold for 340,000, and the loss is large. Training has one goal: make that number small.

Backpropagation works out how much each weight added to the loss. It starts at the output and passes the error backwards, layer by layer. At each weight it gets a number, called the gradient, that says which way to move the weight to lower the loss, and how strongly.

Gradient descent then makes the move. Every weight takes a small step in the direction that lowers the loss. The loss is like a hilly field, and the model is a walker in fog, feeling the slope underfoot and stepping downhill. Many small steps reach a valley.

The learning rate sets the size of each step. Too large, and the walker jumps over the valley and the loss bounces about. Too small, and training crawls. The optimizer is the rule that turns gradients into steps. Adam, from 2014, adjusts the step for each weight as it goes, with a default rate of 0.001.

Training does not use all the data in each step. It splits the rows into batches, often 32 or 64 at a time, and takes one step after each batch. One full pass through all the rows is an epoch. With 1,000 rows and a batch size of 100, one epoch is ten steps.

People think more epochs always help. At first, the loss on new data falls along with the training loss. After a point it starts to rise, which means the network is memorising. More epochs past that point make the model worse, not better.

![A loop of four steps: a forward pass where the network makes a guess, the loss that says how wrong the guess was, backpropagation that traces the error back, and gradient descent that nudges every weight, then repeat.](https://learn.benibix.com/diagrams/nn-training-loop.svg)

### Example: Emma fixes her step size

Emma trains a photo model with a learning rate of 0.5. The loss jumps between 2 and 9 and never settles.

She drops the rate to 0.001 and switches to Adam. Over 20 epochs of 50 batches each, the loss falls smoothly from 2.3 to 0.4.

**Words in this lesson:** [Loss Function](https://learn.benibix.com/term/ai/loss-function), [Backpropagation](https://learn.benibix.com/term/ai/backpropagation), [Gradient Descent](https://learn.benibix.com/term/ai/gradient-descent), [Learning Rate](https://learn.benibix.com/term/ai/learning-rate), [Optimizer (Machine Learning)](https://learn.benibix.com/term/ai/optimizer), [Batch Size](https://learn.benibix.com/term/ai/batch-size), [Epoch](https://learn.benibix.com/term/ai/epoch)

**Question:** What does backpropagation work out?

**Answer:** How much each weight added to the error. It passes the error backwards through the layers and gives each weight a gradient. Gradient descent then uses those to move the weights.

---

Source: https://learn.benibix.com/learn/ai-machine-learning/gradient-descent-and-backprop
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
