Skip to content
BENIBIX Learn

Pretraining vs Fine-Tuning

People often think fine-tuning is where a model learns what it knows. Most of its knowledge comes from pretraining, the huge first round on trillions of tokens. Fine-tuning is a small round on a few thousand examples. It shapes style and skill for one job. It is cheap and quick, but it adds few new facts.

AI💬 LLMs

Pretraining

The first, huge round of training on lots of text. It makes a base model.

Pretraining is the first and biggest round of training for a language model. The model reads a huge amount of text, often trillions of tokens, and learns to guess each next token. It runs for weeks on thousands of GPUs and costs the most by far. What comes out is a base model. It knows a lot, but it does not yet follow requests.

Example
A lab pretrains a model on 15 trillion tokens of web pages, books and code. The run takes two months on 16,000 GPUs.
Why it matters
Almost all of a model's knowledge comes from this step. Later steps mostly shape how it answers.
Full entry
AI💬 LLMs

Fine-Tuning

More training on a model that is done, so it gets good at one job.

Fine-tuning is more training on a model that is already trained, using a small set of examples for one job. The model keeps what it knew and gets better at that job or style. A few hundred to a few thousand good examples are often enough. It is much cheaper than training from scratch. It changes the model's weights, so the new skill is built in.

Example
Lucy's support team fine-tunes a model on 2,000 past replies. Now it answers in the firm's tone and knows its product names.
Why it matters
It is how one general model becomes many special ones. You do not pay for a whole new model.
Full entry

In one line

Pretraining
The first, huge round of training on lots of text. It makes a base model.
Fine-Tuning
More training on a model that is done, so it gets good at one job.