Skip to content
BENIBIX Learn

Encoder vs Decoder

People often think every transformer has both an encoder and a decoder. Many have only one. An encoder reads a whole text, looking both ways. It turns the text into meaning for search or sorting. A decoder writes text one token at a time, looking only back. Chat models use just a decoder.

AI💬 LLMs

Encoder (Transformer)

The part of a transformer that reads text and finds its meaning.

An encoder is the part of a transformer that reads a whole text. It turns the text into vectors that hold its meaning. Each token can look at all the others, both before and after it. An encoder does not write; it is used to sort text, find names and help search. In a translator, the encoder reads the sentence and the decoder writes it out anew.

Example
Emma's search tool uses a model that is only an encoder, a kind made popular by BERT in 2018. It turns each help page into a vector, so a question finds the right page.
Why it matters
Encoders do quiet work behind search and spam filters. They also sort lots of text.
Full entry
AI💬 LLMs

Decoder (Transformer)

The part of a transformer that writes. It adds one token at a time.

A decoder is the part of a transformer that writes text, one token at a time. Each token can only look at the tokens before it. So the model never sees the future while it learns to guess it. Almost every chat model is only a decoder. It is one tall stack of decoder layers. In a translator, the decoder also looks at what the encoder gave it.

Example
Jack asks his chat assistant for a packing list. A model that is only a decoder writes it one token at a time. Each new token looks back at all the ones before.
Why it matters
Decoders are what generate text. Every chat reply you read was written by one.
Full entry

In one line

Encoder (Transformer)
The part of a transformer that reads text and finds its meaning.
Decoder (Transformer)
The part of a transformer that writes. It adds one token at a time.