💬Scaling Laws
Also known as Neural Scaling Laws, LLM Scaling Laws
Rules that say how much better a model gets as it grows.
Think of it like
Think of it like a rule that says how much faster you run if you train more. You can plan ahead.
Example
A lab trains five small models and plots their error. The line tells them a 100B model trained on 2 trillion tokens should reach a set score, before they spend millions.
Why it matters
They are why labs keep building bigger models. The gains were predictable enough to bet on.
Definition
Common mistake
Key takeaways
- 1Error falls with scale. Weights, data, compute.
- 2A straight line. On a ten-times scale.
- 3Plan before spending. Small runs predict big ones.