Transformer Architecture
Level 2 · Inside the model
The design inside almost every LLM. How it turns tokens into lists of numbers, keeps word order, and passes them up through its layers.
Course contentsLesson 5 of 15
Level 1 · What an LLM is
Level 2 · Inside the model
Level 3 · How a model is trained
Level 4 · Choosing and running a model
Level 5 · Where LLMs go wrong
After this lesson you can
- Name the parts of a transformer layer
- Explain why word order needs adding
- Tell encoder, decoder and MoE models apart
13 mincovers 8 glossary entries
Worth reading first: Tokens and Tokenizers
What is inside a transformer
Almost every LLM today is a transformer. The design came from a 2017 paper by researchers at Google Brain. It reads all the tokens in a text at once, not one after another, so it learns far faster on big machines. GPT-3 stacked 96 of its layers, one on top of the next.
First, each token's ID number becomes a vector. A vector is a long list of numbers that stands for the token's meaning. Words used in similar ways end up with similar vectors. From here on, the model only works with these lists of numbers.
A transformer reads every token at the same moment, so on its own it has no sense of order. 'Dog bites man' and 'man bites dog' would look the same. Positional encoding fixes that. It adds a mark of each token's place to its vector, so word order counts.
Then come the layers. Each one does two jobs. Attention lets every token gather what it needs from the other tokens. Then a small network works on each token by itself. The output of one layer is the input to the next. At the top, the last vector becomes a chance for every token.
The first transformer had two halves. The encoder reads the whole input and builds a picture of it. The decoder writes the output one token at a time, and it only sees the tokens before the one it is writing. Translation used both: read the French, then write the English.
Later models split the halves. BERT, from 2018, kept only the encoder, and it is good at sorting and searching text. The GPT models kept only the decoder, and they are good at writing. Most chat models today are decoder-only.
A mixture of experts, or MoE, changes the small network in each layer. It holds many smaller networks, called experts, and a router sends each token to just a few. Mixtral 8x7B has 8 experts per layer and uses 2 per token. It holds about 47 billion parameters but uses about 13 billion per token.
People read 'expert' as a topic, like a maths expert or a history expert. It is not. The router learns its own split, and it often follows word shapes rather than subjects. The win is speed: a big model that does the work of a small one per token, though it still needs memory for all of it.
The three shapes of transformer
Worked example
The words for this lesson8
Check yourself
What does positional encoding tell a model?
Finished this lesson?
Marking it complete also marks its 8 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.