💬Chinchilla Scaling Laws
Also known as Chinchilla, Compute-Optimal Training, Chinchilla Optimal
Train on about 20 words of text for each weight.
Think of it like
It is like a new library. It is no good until you buy books to fill it.
Example
DeepMind's model Chinchilla had 70 billion parameters and read 1.4 trillion tokens, which is 20 for each one. A model with 10 billion would need about 200 billion.
Why it matters
It showed that many big models of the time had read too little. It changed how labs spend their money.
Definition
Common mistake
Key takeaways
- 1Grow both together. Size and data rise as a pair.
- 2About 20 to 1. Tokens of text for each parameter.
- 3Cheapest to train. Not the cheapest to run.