Skip to content
BENIBIX Learn

Gradient Descent

AI📈 Machine LearningNeural Networks

Also known as Stochastic Gradient Descent, SGD

How training improves a model. Each small step nudges the weights downhill.

Think of it like

Think of it like walking down a hill in fog. You feel which way is down and take a step.

Example

Jack's model starts with random weights and a loss of 8.0. After 1,000 steps of gradient descent, the loss is down to 0.3.

Why it matters

It is how almost every neural network learns. Without it, there would be no way to tune billions of weights.

Definition

Gradient descent lowers a model's loss one small step at a time. At each step, it works out which way to nudge every weight so the loss goes down. Then it moves all the weights a little in that direction. It repeats this thousands or millions of times. Most training today uses a version that looks at a small batch of data per step.
Each dot is one step. Small steps reach the bottom. Steps that are too big jump past it.

Common mistake

Taking steps that are too big. The model jumps past the bottom of the hill, and the loss bounces or grows.

Key takeaways

  1. 1
    Find downhill. Which way lowers the loss.
  2. 2
    Take a small step. Nudge all weights a little.
  3. 3
    Repeat. Thousands or millions of times.

Compare

Learn it properly