Pretraining an LLM
Level 3 · How a model is trained
How a model learns from trillions of tokens of text. What scaling laws predict, and why the result does not follow requests yet.
Course contentsLesson 7 of 15
Level 1 · What an LLM is
Level 2 · Inside the model
Level 3 · How a model is trained
Level 4 · Choosing and running a model
Level 5 · Where LLMs go wrong
After this lesson you can
- Describe what a model practises in pretraining
- Explain what scaling laws predict
- Say why a base model ignores requests
12 mincovers 5 glossary entries
Worth reading first: Tokens and Tokenizers
How a model learns from trillions of tokens
Pretraining is the first and biggest stage of making an LLM. The model reads a huge amount of text and practises one task: guessing the next token. Meta's Llama 3 read over 15 trillion tokens. A run like that takes weeks on thousands of chips and costs millions.
That text is the training data. It is mostly web pages, with books, code, papers and forum posts mixed in. Before use it is cleaned: copies are removed, spam is thrown out, and the mix is balanced. The mix shapes the model as much as its size does.
Learning works by being wrong. The model guesses the next token, sees the real one, and every parameter is nudged a tiny step towards a better guess. That happens trillions of times. To guess well across all that text, the model has to pick up grammar, facts and some reasoning on the way.
Researchers found that this gets better in a steady way you can predict. Each time a lab gives a model ten times the compute, its error drops by about the same step. These patterns are called scaling laws. They let a lab test small models first and predict how a big one will do.
Scaling laws also say how to spend the budget. In 2022, DeepMind's Chinchilla, with 70 billion parameters, read 1.4 trillion tokens. It beat their earlier model, which was four times bigger but read far less text. The rough rule became about 20 tokens of text for every parameter.
Some skills seem to appear all at once. A small model fails a kind of sum every time, and a larger one suddenly gets it right. These are called emergent abilities. Later work showed part of the jump comes from the scoring: marking answers only fully right or wrong hides slow, steady gains.
What comes out of pretraining is a base model. It is very good at carrying on text, but it does not follow requests. Ask it a question and it may write five more questions, because that is how a lot of the text it read looks. Making it helpful is the next stage.
Worked example
Lucy plans a training run
Lucy's team wants a model with 1 billion parameters. The rule is about 20 tokens for each one. So it should read about 20 billion tokens.
Their cleaned text holds only 5 billion tokens. So they either gather four times more text, or plan a smaller model.
The words for this lesson5
Check yourself
What does an LLM practise during pretraining?
Finished this lesson?
Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.