💬Pretraining
Also known as Pre-training, LLM Pretraining
The first, huge round of training on lots of text. It makes a base model.
Think of it like
Think of it like a kid who reads all the books in town. Only then does school start.
Example
A lab pretrains a model on 15 trillion tokens of web pages, books and code. The run takes two months on 16,000 GPUs.
Why it matters
Almost all of a model's knowledge comes from this step. Later steps mostly shape how it answers.
Definition
Common mistake
Key takeaways
- 1First and biggest. Weeks on thousands of GPUs.
- 2Guess the next token. Over trillions of them.
- 3Makes a base model. Knows a lot, follows nothing.