Pretraining vs Fine-Tuning
People often think fine-tuning is where a model learns what it knows. Most of its knowledge comes from pretraining, the huge first round on trillions of tokens. Fine-tuning is a small round on a few thousand examples. It shapes style and skill for one job. It is cheap and quick, but it adds few new facts.
Pretraining
The first, huge round of training on lots of text. It makes a base model.
Pretraining is the first and biggest round of training for a language model. The model reads a huge amount of text, often trillions of tokens, and learns to guess each next token. It runs for weeks on thousands of GPUs and costs the most by far. What comes out is a base model. It knows a lot, but it does not yet follow requests.
Fine-Tuning
More training on a model that is done, so it gets good at one job.
Fine-tuning is more training on a model that is already trained, using a small set of examples for one job. The model keeps what it knew and gets better at that job or style. A few hundred to a few thousand good examples are often enough. It is much cheaper than training from scratch. It changes the model's weights, so the new skill is built in.
In one line
- Pretraining
- The first, huge round of training on lots of text. It makes a base model.
- Fine-Tuning
- More training on a model that is done, so it gets good at one job.