Skip to content
BENIBIX Learn

Optimizer (Machine Learning)

AI📈 Machine LearningNeural Networks

Also known as Adam Optimizer, Adam, Optimization Algorithm (Neural Networks)

The rule for how a network moves its weights at each step as it learns.

Think of it like

It is like a way to walk down a hill in fog. Some walk slow. Some speed up.

Example

Emma trains an image model with plain gradient descent, and it crawls. She switches to Adam, and the loss drops in half the epochs.

Why it matters

The optimizer decides how fast and how well a network learns. A poor choice can stall training.

Definition

An optimizer is the rule a network uses to change its weights while it trains. It reads the gradient, which says which way lowers the loss. Then it decides how big a step to take, and in which direction. Plain gradient descent takes the same size step each time. Adam, the most common one, adjusts the step for each weight as it goes. A good optimizer reaches a low loss in fewer steps.

Example code

python
import torch
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss.backward()
optimizer.step()
It reads which way is down, then moves each weight one step.
An optimizer picks each step down the curve. Steps that are too big jump past the bottom.

Common mistake

Changing the optimizer but keeping the old learning rate. Each one wants its own rate.

Key takeaways

  1. 1
    Reads the gradient. Which way lowers the loss.
  2. 2
    Picks the step. How big, and for which weights.
  3. 3
    Adam is common. It adapts the step for each weight.

Learn it properly