Skip to content
BENIBIX Learn

Knowledge Distillation

AI💬 LLMsLanguage Model Training

Also known as Model Distillation, Distillation

A small model is trained to copy a big one. It is fast and still good.

Think of it like

Think of it like a teacher who shows a kid how they think. The kid learns fast that way.

Example

A firm uses its 400B model to answer a million questions. A 7B student learns from those answers and runs ten times cheaper.

Why it matters

It is how much of the skill in huge models ends up in small ones that run on phones and laptops.

Definition

Knowledge distillation trains a small model, the student, to copy a large model, the teacher. The student learns from the teacher's answers and from how sure the teacher was about each word. That carries more than right or wrong alone. The student ends up much smaller and faster, and keeps much of the teacher's skill. Many small models are made this way.
The student learns from the teacher's answers and from how sure it was about each word.

Common mistake

Expecting the student to match the teacher. It keeps much of the skill, not all, and it can copy the teacher's mistakes too.

Key takeaways

  1. 1
    Teacher and student. Big model, small model.
  2. 2
    Copies how sure. Not just right or wrong.
  3. 3
    Smaller and faster. Keeps much of the skill.

Learn it properly