Skip to content
BENIBIX Learn

Encoder (Transformer)

AI💬 LLMsTransformer Architecture

Also known as Encoder, Encoder-Only Model

The part of a transformer that reads text and finds its meaning.

Think of it like

Think of it like a reader who reads the whole page. Then they tell you what it means.

Example

Emma's search tool uses a model that is only an encoder, a kind made popular by BERT in 2018. It turns each help page into a vector, so a question finds the right page.

Why it matters

Encoders do quiet work behind search and spam filters. They also sort lots of text.

Definition

An encoder is the part of a transformer that reads a whole text. It turns the text into vectors that hold its meaning. Each token can look at all the others, both before and after it. An encoder does not write; it is used to sort text, find names and help search. In a translator, the encoder reads the sentence and the decoder writes it out anew.
The encoder, on the left, reads the text and hands back its meaning, not new text.

Common mistake

Expecting an encoder-only model to chat. It reads and understands, but it has no part that writes.

Key takeaways

  1. 1
    Reads the whole text. Both directions.
  2. 2
    Gives vectors. The meaning as numbers.
  3. 3
    Does not write. Used for search and sorting.

Compare

Learn it properly