🔤DistilBERT
Also known as Distilled BERT
A smaller, faster BERT that kept 97% of its skill.
Think of it like
It is like a short copy of a long book. It keeps nearly all the story in far fewer pages.
Example
Jack's app sorts support emails by topic. He downloads DistilBERT from Hugging Face, and it runs fast on a plain server.
Why it matters
It was an early sign that a model can shrink a lot and still lose very little.
Definition
Common mistake
Key takeaways
- 1Learned from BERT. BERT was the teacher, DistilBERT the student.
- 240% smaller. 66 million parameters.
- 3Reads, does not write. Good at sorting and tagging.