# Knowledge Distillation

> How a big model teaches a small one, and what small models are good at. The risk of training a model on text that models wrote.

**Course:** [Large Language Models (LLMs)](https://learn.benibix.com/learn/ai-llms)  
**Time:** 12 min

**After this lesson you can:**

- Explain how a student model learns from a teacher
- Say when a small model is the better pick
- Describe model collapse and its fix

## How a big model teaches a small one

Knowledge distillation trains a small model to copy a big one. The big model, the teacher, answers many questions. The small one, the student, learns to match those answers. In 2019, DistilBERT came out 40% smaller and 60% faster than its teacher, and kept 97% of its skill on language tests.

The trick is that the teacher gives more than one answer. It gives its chance for every token. A student that learns 'most likely cat, quite likely dog, never car' learns more from one example than from the word 'cat' alone. Today, many teams just train on the teacher's written answers.

The students are often small language models, or SLMs. There is no strict line, but it usually means a few billion parameters or fewer. The Phi-3-mini model has 3.8 billion. A model that size runs on a phone or a laptop, answers fast and costs little.

Small models win on narrow jobs. One trained only to sort support emails can match a big model on that task, for a small part of the cost. They fall behind on wide knowledge and long chains of reasoning. There are fewer parameters to hold it all.

Distillation is one use of synthetic data: text written by a model to train a model. Labs use it to make practice questions, to fill gaps in rare languages and to build sets of worked maths. Good use needs a filter. Write many, check them, keep only the good ones.

Synthetic data has a known failure. Train a model on another model's output, then train the next one on that, and so on. A 2024 study showed that each round loses rare and unusual cases, and the output gets narrower and blander. This is model collapse.

The fix is to keep real, human-written text in the mix and to filter what goes in. As more of the web is written by models, labs work harder to find and mark text that is real. Old sources that are clearly human have become more valuable as a result.

People assume a small model is a cut-down copy that does everything worse. On its one job, a well-made small model can match a giant, and run faster. The question is not big or small. It is how wide the job is.

![A large box labelled teacher, a large model, sends its answers and how sure it is to a smaller box labelled student, a small, fast model, which learns to match them.](https://learn.benibix.com/diagrams/llm-distillation.svg)

### Example: Lucy shrinks her email sorter

Lucy uses a large model to label 50,000 support emails by topic. She then trains a model with 1 billion parameters on those labels.

On her test set the small model sorts new emails as well as the large one. It runs on her own server, in a tenth of the time.

**Words in this lesson:** [Knowledge Distillation](https://learn.benibix.com/term/ai/knowledge-distillation), [Small Language Model (SLM)](https://learn.benibix.com/term/ai/small-language-model), [Synthetic Data](https://learn.benibix.com/term/ai/synthetic-data), [Model Collapse](https://learn.benibix.com/term/ai/model-collapse), [DistilBERT](https://learn.benibix.com/term/ai/distilbert)

**Question:** What is model collapse?

**Answer:** Output getting narrower after rounds of training on model-written text. Each round of training on another model's output loses some rare cases. Over many rounds the text gets narrow and bland. Real human text in the mix stops it.

---

Source: https://learn.benibix.com/learn/ai-llms/knowledge-distillation
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
