Skip to content
BENIBIX Learn

Pretraining

AI💬 LLMsLanguage Model Training

Also known as Pre-training, LLM Pretraining

The first, huge round of training on lots of text. It makes a base model.

Think of it like

Think of it like a kid who reads all the books in town. Only then does school start.

Example

A lab pretrains a model on 15 trillion tokens of web pages, books and code. The run takes two months on 16,000 GPUs.

Why it matters

Almost all of a model's knowledge comes from this step. Later steps mostly shape how it answers.

Definition

Pretraining is the first and biggest round of training for a language model. The model reads a huge amount of text, often trillions of tokens, and learns to guess each next token. It runs for weeks on thousands of GPUs and costs the most by far. What comes out is a base model. It knows a lot, but it does not yet follow requests.
Pretraining is the first and largest stage. The later stages are much smaller.

Common mistake

Thinking pretraining teaches the model to chat. It only teaches it to carry text on.

Key takeaways

  1. 1
    First and biggest. Weeks on thousands of GPUs.
  2. 2
    Guess the next token. Over trillions of them.
  3. 3
    Makes a base model. Knows a lot, follows nothing.

Compare

Learn it properly