# Optimizer (Machine Learning)

> The rule for how a network moves its weights at each step as it learns.

**Field:** Artificial Intelligence  
**Category:** Machine Learning  
**Also known as:** Adam Optimizer, Adam, Optimization Algorithm (Neural Networks)

## Think of it like

It is like a way to walk down a hill in fog. Some walk slow. Some speed up.

## Definition

An optimizer is the rule a network uses to change its weights while it trains. It reads the gradient, which says which way lowers the loss. Then it decides how big a step to take, and in which direction. Plain gradient descent takes the same size step each time. Adam, the most common one, adjusts the step for each weight as it goes. A good optimizer reaches a low loss in fewer steps.

## Example

Emma trains an image model with plain gradient descent, and it crawls. She switches to Adam, and the loss drops in half the epochs.

## Example code

```python
import torch
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss.backward()
optimizer.step()
```

It reads which way is down, then moves each weight one step.

## Why it matters

The optimizer decides how fast and how well a network learns. A poor choice can stall training.

## Common mistake

Changing the optimizer but keeping the old learning rate. Each one wants its own rate.

## Three principles

- **Reads the gradient.** Which way lowers the loss.
- **Picks the step.** How big, and for which weights.
- **Adam is common.** It adapts the step for each weight.

## Related terms

- [Gradient Descent](https://learn.benibix.com/term/ai/gradient-descent)
- [Learning Rate](https://learn.benibix.com/term/ai/learning-rate)
- [Backpropagation](https://learn.benibix.com/term/ai/backpropagation)
- [Loss Function](https://learn.benibix.com/term/ai/loss-function)
- [Weight (Neural Network)](https://learn.benibix.com/term/ai/weight)

## Taught in

- [Gradient Descent and Backprop](https://learn.benibix.com/learn/ai-machine-learning/gradient-descent-and-backprop): Machine Learning, 14/18

![A U-shaped curve of loss. A ball takes small steps down the slope to the bottom. On a second curve, steps that are too big jump from side to side past the bottom.](https://learn.benibix.com/diagrams/nn-gradient-descent.svg)

---

Source: https://learn.benibix.com/term/ai/optimizer
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
