# Knowledge Distillation

> A small model is trained to copy a big one. It is fast and still good.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** Model Distillation, Distillation

## Think of it like

Think of it like a teacher who shows a kid how they think. The kid learns fast that way.

## Definition

Knowledge distillation trains a small model, the student, to copy a large model, the teacher. The student learns from the teacher's answers and from how sure the teacher was about each word. That carries more than right or wrong alone. The student ends up much smaller and faster, and keeps much of the teacher's skill. Many small models are made this way.

## Example

A firm uses its 400B model to answer a million questions. A 7B student learns from those answers and runs ten times cheaper.

## Why it matters

It is how much of the skill in huge models ends up in small ones that run on phones and laptops.

## Common mistake

Expecting the student to match the teacher. It keeps much of the skill, not all, and it can copy the teacher's mistakes too.

## Three principles

- **Teacher and student.** Big model, small model.
- **Copies how sure.** Not just right or wrong.
- **Smaller and faster.** Keeps much of the skill.

## Related terms

- [Small Language Model (SLM)](https://learn.benibix.com/term/ai/small-language-model)
- [Synthetic Data](https://learn.benibix.com/term/ai/synthetic-data)
- [Parameter Count](https://learn.benibix.com/term/ai/parameter-count)
- [Quantization (AI)](https://learn.benibix.com/term/ai/quantization)

## Taught in

- [Knowledge Distillation](https://learn.benibix.com/learn/ai-llms/knowledge-distillation): LLMs, 10/15

![A large teacher model gives its answers and its chances for each word. A small student model is trained to match them.](https://learn.benibix.com/diagrams/llm-distillation.svg)

---

Source: https://learn.benibix.com/term/ai/knowledge-distillation
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
