Skip to content
BENIBIX Learn

DistilBERT

AI🔤 NLP & LanguageEncoder and Decoder Models

Also known as Distilled BERT

A smaller, faster BERT that kept 97% of its skill.

Think of it like

It is like a short copy of a long book. It keeps nearly all the story in far fewer pages.

Example

Jack's app sorts support emails by topic. He downloads DistilBERT from Hugging Face, and it runs fast on a plain server.

Why it matters

It was an early sign that a model can shrink a lot and still lose very little.

Definition

DistilBERT is a smaller copy of the BERT language model, built to sort and tag text fast. It was made in 2019 through knowledge distillation. It has 66 million parameters, against 110 million in the BERT it learned from. It is 40% smaller and 60% faster. It keeps about 97% of BERT's score on language tests.
BERT was the teacher here, and DistilBERT the student.

Common mistake

Using DistilBERT to write text. Like BERT, it reads and sorts text. It does not write replies.

Key takeaways

  1. 1
    Learned from BERT. BERT was the teacher, DistilBERT the student.
  2. 2
    40% smaller. 66 million parameters.
  3. 3
    Reads, does not write. Good at sorting and tagging.

Learn it properly