Encoder vs Decoder
People often think every transformer has both an encoder and a decoder. Many have only one. An encoder reads a whole text, looking both ways. It turns the text into meaning for search or sorting. A decoder writes text one token at a time, looking only back. Chat models use just a decoder.
Encoder (Transformer)
The part of a transformer that reads text and finds its meaning.
An encoder is the part of a transformer that reads a whole text. It turns the text into vectors that hold its meaning. Each token can look at all the others, both before and after it. An encoder does not write; it is used to sort text, find names and help search. In a translator, the encoder reads the sentence and the decoder writes it out anew.
Decoder (Transformer)
The part of a transformer that writes. It adds one token at a time.
A decoder is the part of a transformer that writes text, one token at a time. Each token can only look at the tokens before it. So the model never sees the future while it learns to guess it. Almost every chat model is only a decoder. It is one tall stack of decoder layers. In a translator, the decoder also looks at what the encoder gave it.
In one line
- Encoder (Transformer)
- The part of a transformer that reads text and finds its meaning.
- Decoder (Transformer)
- The part of a transformer that writes. It adds one token at a time.