# Self-Attention and Multi-Head

> How each token pulls meaning from the others. Why long texts cost so much, and what many heads and faster maths change.

**Course:** [Large Language Models (LLMs)](https://learn.benibix.com/learn/ai-llms)  
**Time:** 13 min

**After this lesson you can:**

- Explain how a token picks what to look at
- Say why long inputs are slow
- Describe what each attention head does

## How each token decides what to look at

Self-attention is how each token in a text gathers meaning from the others. For every token, the model scores how much each other token matters to it. Then it mixes in what it finds. In 'the animal did not cross the street because it was tired', the word 'it' pulls most from 'animal'.

Each token makes three things from its vector: a question, a label and a message. Its question is checked against every other token's label. A close match gives a high score. The token then takes in each message in line with its score. That is the whole trick, and it runs in every layer.

There is a catch. Every token scores every other token. So 1,000 tokens means about a million scores, and 2,000 tokens means about four million. Double the text and the work goes up four times. This is a big reason long inputs are slow and dear.

One set of scores can only follow one kind of link at a time. Multi-head attention runs several sets side by side. The first transformer used 8 heads. Each head learns its own pattern. One might link a word to the word before it. Another might link a verb to who did it.

The heads work at the same time, and their results are joined back into one vector. Nobody tells a head what to look for. Researchers who study trained models do find heads with clear jobs, but many do things that are hard to name.

FlashAttention is a faster way to do the same maths. The slow part of attention is not the sums. It is moving huge tables of scores in and out of the chip's main memory. FlashAttention works in small tiles that fit in the chip's fast memory, so the big tables are never written out whole.

The answer is exactly the same, only quicker. Its makers measured GPT-2 learning about 3 times faster with it. It also needs far less memory for long inputs, which is part of why context windows grew so much after 2022.

A common belief is that attention shows why a model gave an answer. The scores show where each token took its facts from, in one layer and one head. With dozens of layers and heads all mixing, a picture of one of them explains far less than it seems to.

![The sentence 'The animal didn't cross the street because it was tired' with lines from the word 'it' to the other words. The line to 'animal' is the thickest; the line to 'street' is thin. A note says a thick line means strong attention.](https://learn.benibix.com/diagrams/llm-attention.svg)

### Example: Jack sends a longer contract

Jack sends a 4,000-token contract to a model, then a 16,000-token one. The second file is 4 times longer.

Inside each layer, the attention work grows about 16 times. That is why the bigger file takes so much longer before the first word of the answer appears.

**Words in this lesson:** [Self-Attention](https://learn.benibix.com/term/ai/self-attention), [Multi-Head Attention](https://learn.benibix.com/term/ai/multi-head-attention), [FlashAttention](https://learn.benibix.com/term/ai/flash-attention)

**Question:** Why does doubling a text make attention about four times the work?

**Answer:** Every token scores every other token. Each token is scored against every other one. Twice the tokens gives twice the rows and twice the columns of scores, so four times as many in all.

---

Source: https://learn.benibix.com/learn/ai-llms/self-attention-and-multi-head
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
