💬Knowledge Distillation
Also known as Model Distillation, Distillation
A small model is trained to copy a big one. It is fast and still good.
Think of it like
Think of it like a teacher who shows a kid how they think. The kid learns fast that way.
Example
A firm uses its 400B model to answer a million questions. A 7B student learns from those answers and runs ten times cheaper.
Why it matters
It is how much of the skill in huge models ends up in small ones that run on phones and laptops.
Definition
Common mistake
Key takeaways
- 1Teacher and student. Big model, small model.
- 2Copies how sure. Not just right or wrong.
- 3Smaller and faster. Keeps much of the skill.