# Pretraining

> The first, huge round of training on lots of text. It makes a base model.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** Pre-training, LLM Pretraining

## Think of it like

Think of it like a kid who reads all the books in town. Only then does school start.

## Definition

Pretraining is the first and biggest round of training for a language model. The model reads a huge amount of text, often trillions of tokens, and learns to guess each next token. It runs for weeks on thousands of GPUs and costs the most by far. What comes out is a base model. It knows a lot, but it does not yet follow requests.

## Example

A lab pretrains a model on 15 trillion tokens of web pages, books and code. The run takes two months on 16,000 GPUs.

## Why it matters

Almost all of a model's knowledge comes from this step. Later steps mostly shape how it answers.

## Common mistake

Thinking pretraining teaches the model to chat. It only teaches it to carry text on.

## Three principles

- **First and biggest.** Weeks on thousands of GPUs.
- **Guess the next token.** Over trillions of them.
- **Makes a base model.** Knows a lot, follows nothing.

## Related terms

- [Fine-Tuning](https://learn.benibix.com/term/ai/fine-tuning)
- [Base Model](https://learn.benibix.com/term/ai/base-model)
- [Next-Token Prediction](https://learn.benibix.com/term/ai/next-token-prediction)
- [Training (AI)](https://learn.benibix.com/term/ai/training)
- [Post-Training](https://learn.benibix.com/term/ai/post-training)

## Told apart from

- [Pretraining vs Fine-Tuning](https://learn.benibix.com/compare/pretraining-vs-fine-tuning): People often think fine-tuning is where a model learns what it knows. Most of its knowledge comes from pretraining, the huge first round on trillions of tokens. Fine-tuning is a small round on a few thousand examples. It shapes style and skill for one job. It is cheap and quick, but it adds few new facts.

## Taught in

- [Pretraining an LLM](https://learn.benibix.com/learn/ai-llms/pretraining-an-llm): LLMs, 7/15

![Three stages from left to right. Pretraining on trillions of tokens makes a base model. Instruction tuning on many examples makes it follow requests. Feedback from people makes it a helpful chat model.](https://learn.benibix.com/diagrams/llm-training-stages.svg)

---

Source: https://learn.benibix.com/term/ai/pretraining
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
