# Gradient Descent

> How training improves a model. Each small step nudges the weights downhill.

**Field:** Artificial Intelligence  
**Category:** Machine Learning  
**Also known as:** Stochastic Gradient Descent, SGD

## Think of it like

Think of it like walking down a hill in fog. You feel which way is down and take a step.

## Definition

Gradient descent lowers a model's loss one small step at a time. At each step, it works out which way to nudge every weight so the loss goes down. Then it moves all the weights a little in that direction. It repeats this thousands or millions of times. Most training today uses a version that looks at a small batch of data per step.

## Example

Jack's model starts with random weights and a loss of 8.0. After 1,000 steps of gradient descent, the loss is down to 0.3.

## Why it matters

It is how almost every neural network learns. Without it, there would be no way to tune billions of weights.

## Common mistake

Taking steps that are too big. The model jumps past the bottom of the hill, and the loss bounces or grows.

## Three principles

- **Find downhill.** Which way lowers the loss.
- **Take a small step.** Nudge all weights a little.
- **Repeat.** Thousands or millions of times.

## Related terms

- [Loss Function](https://learn.benibix.com/term/ai/loss-function)
- [Learning Rate](https://learn.benibix.com/term/ai/learning-rate)
- [Backpropagation](https://learn.benibix.com/term/ai/backpropagation)
- [Batch Size](https://learn.benibix.com/term/ai/batch-size)
- [Training (AI)](https://learn.benibix.com/term/ai/training)
- [Gradient](https://learn.benibix.com/term/general/gradient)
- [Derivative](https://learn.benibix.com/term/general/derivative)

## Told apart from

- [Backpropagation vs Gradient Descent](https://learn.benibix.com/compare/backpropagation-vs-gradient-descent): People often use the two names for the same thing. They are two steps that work as a pair. Backpropagation works out how much each weight added to the error. Gradient descent then uses those numbers to move every weight a small step. One finds the changes, and the other makes them.

## Taught in

- [Gradient Descent and Backprop](https://learn.benibix.com/learn/ai-machine-learning/gradient-descent-and-backprop): Machine Learning, 14/18

![A U-shaped curve of loss. A ball takes small steps down the slope to the bottom. On a second curve, steps that are too big jump from side to side past the bottom.](https://learn.benibix.com/diagrams/nn-gradient-descent.svg)

---

Source: https://learn.benibix.com/term/ai/gradient-descent
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
