Self-Attention and Multi-Head
Level 2 · Inside the model
How each token pulls meaning from the others. Why long texts cost so much, and what many heads and faster maths change.
Course contentsLesson 6 of 15
Level 1 · What an LLM is
Level 2 · Inside the model
Level 3 · How a model is trained
Level 4 · Choosing and running a model
Level 5 · Where LLMs go wrong
After this lesson you can
- Explain how a token picks what to look at
- Say why long inputs are slow
- Describe what each attention head does
13 mincovers 3 glossary entries
Worth reading first: Transformer Architecture
How each token decides what to look at
Self-attention is how each token in a text gathers meaning from the others. For every token, the model scores how much each other token matters to it. Then it mixes in what it finds. In 'the animal did not cross the street because it was tired', the word 'it' pulls most from 'animal'.
Each token makes three things from its vector: a question, a label and a message. Its question is checked against every other token's label. A close match gives a high score. The token then takes in each message in line with its score. That is the whole trick, and it runs in every layer.
There is a catch. Every token scores every other token. So 1,000 tokens means about a million scores, and 2,000 tokens means about four million. Double the text and the work goes up four times. This is a big reason long inputs are slow and dear.
One set of scores can only follow one kind of link at a time. Multi-head attention runs several sets side by side. The first transformer used 8 heads. Each head learns its own pattern. One might link a word to the word before it. Another might link a verb to who did it.
The heads work at the same time, and their results are joined back into one vector. Nobody tells a head what to look for. Researchers who study trained models do find heads with clear jobs, but many do things that are hard to name.
FlashAttention is a faster way to do the same maths. The slow part of attention is not the sums. It is moving huge tables of scores in and out of the chip's main memory. FlashAttention works in small tiles that fit in the chip's fast memory, so the big tables are never written out whole.
The answer is exactly the same, only quicker. Its makers measured GPT-2 learning about 3 times faster with it. It also needs far less memory for long inputs, which is part of why context windows grew so much after 2022.
A common belief is that attention shows why a model gave an answer. The scores show where each token took its facts from, in one layer and one head. With dozens of layers and heads all mixing, a picture of one of them explains far less than it seems to.
Worked example
Jack sends a longer contract
Jack sends a 4,000-token contract to a model, then a 16,000-token one. The second file is 4 times longer.
Inside each layer, the attention work grows about 16 times. That is why the bigger file takes so much longer before the first word of the answer appears.
The words for this lesson3
Check yourself
Why does doubling a text make attention about four times the work?
Finished this lesson?
Marking it complete also marks its 3 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.