Skip to content
BENIBIX Learn

Chinchilla Scaling Laws

AI💬 LLMsLanguage Model Training

Also known as Chinchilla, Compute-Optimal Training, Chinchilla Optimal

Train on about 20 words of text for each weight.

Think of it like

It is like a new library. It is no good until you buy books to fill it.

Example

DeepMind's model Chinchilla had 70 billion parameters and read 1.4 trillion tokens, which is 20 for each one. A model with 10 billion would need about 200 billion.

Why it matters

It showed that many big models of the time had read too little. It changed how labs spend their money.

Definition

The Chinchilla scaling laws are a rule for splitting a training budget between model size and data. They were published in 2022. They say a model should read about 20 tokens of text for each parameter. A model trained this way beat one four times its size that had read far less. Since then, labs train smaller models on much more text.
Chinchilla says how to spend each step of compute. Put about half into a bigger model and half into more text.

Common mistake

Treating 20 tokens per parameter as the best model to ship. It is the cheapest way to train. Labs now go far past it, because a smaller model is cheaper to run.

Key takeaways

  1. 1
    Grow both together. Size and data rise as a pair.
  2. 2
    About 20 to 1. Tokens of text for each parameter.
  3. 3
    Cheapest to train. Not the cheapest to run.

Learn it properly