Knowledge Distillation
Level 3 · How a model is trained
How a big model teaches a small one, and what small models are good at. The risk of training a model on text that models wrote.
Course contentsLesson 10 of 15
Level 1 · What an LLM is
Level 2 · Inside the model
Level 3 · How a model is trained
Level 4 · Choosing and running a model
Level 5 · Where LLMs go wrong
After this lesson you can
- Explain how a student model learns from a teacher
- Say when a small model is the better pick
- Describe model collapse and its fix
12 mincovers 5 glossary entries
Worth reading first: Pretraining an LLM
How a big model teaches a small one
Knowledge distillation trains a small model to copy a big one. The big model, the teacher, answers many questions. The small one, the student, learns to match those answers. In 2019, DistilBERT came out 40% smaller and 60% faster than its teacher, and kept 97% of its skill on language tests.
The trick is that the teacher gives more than one answer. It gives its chance for every token. A student that learns 'most likely cat, quite likely dog, never car' learns more from one example than from the word 'cat' alone. Today, many teams just train on the teacher's written answers.
The students are often small language models, or SLMs. There is no strict line, but it usually means a few billion parameters or fewer. The Phi-3-mini model has 3.8 billion. A model that size runs on a phone or a laptop, answers fast and costs little.
Small models win on narrow jobs. One trained only to sort support emails can match a big model on that task, for a small part of the cost. They fall behind on wide knowledge and long chains of reasoning. There are fewer parameters to hold it all.
Distillation is one use of synthetic data: text written by a model to train a model. Labs use it to make practice questions, to fill gaps in rare languages and to build sets of worked maths. Good use needs a filter. Write many, check them, keep only the good ones.
Synthetic data has a known failure. Train a model on another model's output, then train the next one on that, and so on. A 2024 study showed that each round loses rare and unusual cases, and the output gets narrower and blander. This is model collapse.
The fix is to keep real, human-written text in the mix and to filter what goes in. As more of the web is written by models, labs work harder to find and mark text that is real. Old sources that are clearly human have become more valuable as a result.
People assume a small model is a cut-down copy that does everything worse. On its one job, a well-made small model can match a giant, and run faster. The question is not big or small. It is how wide the job is.
Worked example
Lucy shrinks her email sorter
Lucy uses a large model to label 50,000 support emails by topic. She then trains a model with 1 billion parameters on those labels.
On her test set the small model sorts new emails as well as the large one. It runs on her own server, in a tenth of the time.
The words for this lesson5
Check yourself
What is model collapse?
Finished this lesson?
Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.