Skip to content
BENIBIX Learn

Scaling Laws

AI💬 LLMsLanguage Model Training

Also known as Neural Scaling Laws, LLM Scaling Laws

Rules that say how much better a model gets as it grows.

Think of it like

Think of it like a rule that says how much faster you run if you train more. You can plan ahead.

Example

A lab trains five small models and plots their error. The line tells them a 100B model trained on 2 trillion tokens should reach a set score, before they spend millions.

Why it matters

They are why labs keep building bigger models. The gains were predictable enough to bet on.

Definition

Scaling laws are patterns that show how a model's error falls as it gets more weights, data and compute. On a chart where each step is ten times bigger, the error falls along a nearly straight line. So a lab can predict how good a big model will be from small test runs. A known 2022 result found about 20 training tokens per weight works best for a set budget.
Each step to the right is ten times more compute. The error drops by about the same amount each step.

Common mistake

Thinking the line goes on forever with no cost. Each gain needs about ten times more compute than the last.

Key takeaways

  1. 1
    Error falls with scale. Weights, data, compute.
  2. 2
    A straight line. On a ten-times scale.
  3. 3
    Plan before spending. Small runs predict big ones.

Learn it properly