Skip to content
BENIBIX Learn

Learning Rate

AI📈 Machine LearningNeural Networks

Also known as Step Size

How big each training step is. Too big jumps around, too small crawls.

Think of it like

Think of it like the size of your steps down a steep hill. Too big and you trip.

Example

Emma trains with a learning rate of 0.5, and the loss bounces wildly. She drops it to 0.001, and the loss falls smoothly.

Why it matters

It is often the single most important setting to tune. The wrong value can waste days of training.

Definition

The learning rate is a setting that decides how big each training step is. With a large learning rate, the weights change a lot at each step. With a small one, they change only a little. Too large, and training jumps around and never settles. Too small, and training takes far too long. Common values are small, like 0.001.
The left curve has a good learning rate. On the right, it is too big, and the steps bounce past the bottom.

Common mistake

Keeping it the same from start to finish. Many teams start larger and shrink it as training goes on.

Key takeaways

  1. 1
    Step size. How far each update moves.
  2. 2
    Too big. Jumps and never settles.
  3. 3
    Too small. Slow, and may stall.

Learn it properly