# DistilBERT

> A smaller, faster BERT that kept 97% of its skill.

**Field:** Artificial Intelligence  
**Category:** NLP & Language  
**Also known as:** Distilled BERT

## Think of it like

It is like a short copy of a long book. It keeps nearly all the story in far fewer pages.

## Definition

DistilBERT is a smaller copy of the BERT language model, built to sort and tag text fast. It was made in 2019 through knowledge distillation. It has 66 million parameters, against 110 million in the BERT it learned from. It is 40% smaller and 60% faster. It keeps about 97% of BERT's score on language tests.

## Example

Jack's app sorts support emails by topic. He downloads DistilBERT from Hugging Face, and it runs fast on a plain server.

## Why it matters

It was an early sign that a model can shrink a lot and still lose very little.

## Common mistake

Using DistilBERT to write text. Like BERT, it reads and sorts text. It does not write replies.

## Three principles

- **Learned from BERT.** BERT was the teacher, DistilBERT the student.
- **40% smaller.** 66 million parameters.
- **Reads, does not write.** Good at sorting and tagging.

## Related terms

- [BERT (Language Model)](https://learn.benibix.com/term/ai/bert)
- [Knowledge Distillation](https://learn.benibix.com/term/ai/knowledge-distillation)
- [Small Language Model (SLM)](https://learn.benibix.com/term/ai/small-language-model)
- [Encoder (Transformer)](https://learn.benibix.com/term/ai/encoder)
- [Hugging Face](https://learn.benibix.com/term/ai/hugging-face)

## Taught in

- [Knowledge Distillation](https://learn.benibix.com/learn/ai-llms/knowledge-distillation): LLMs, 10/15

![A large teacher model gives its answers and its chances for each word. A small student model is trained to match them.](https://learn.benibix.com/diagrams/llm-distillation.svg)

---

Source: https://learn.benibix.com/term/ai/distilbert
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
