# Multi-Head Attention

> Many attention steps run side by side. Each finds its own links.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** Multi-Head Self-Attention, Attention Heads, Attention Head

## Think of it like

It is like a team of readers. One looks for names, one for times, and one for places.

## Definition

Multi-head attention runs several attention steps side by side on the same text. Each one, called a head, has its own learned weights, so it can notice a different kind of link between tokens. One head may track which name a word like 'she' points to, and another may track word order. The outputs of all the heads are joined and mixed into one result. The first transformer, in 2017, used 8 heads in each attention layer.

## Example

In the line 'Sam gave Lily a book because she asked', one head links 'she' to 'Lily'. Another head links 'gave' to 'book'.

## Why it matters

One head can only weigh the tokens one way. Many heads let a model see many kinds of links at once.

## Common mistake

Thinking a person gives each head its job. The jobs are learned in training, and many heads are hard to explain.

## Three principles

- **Many heads.** Run side by side.
- **Own weights.** Each looks for its own links.
- **Joined at the end.** Mixed into one output.

## Related terms

- [Attention Mechanism](https://learn.benibix.com/term/ai/attention-mechanism)
- [Self-Attention](https://learn.benibix.com/term/ai/self-attention)
- [Transformer](https://learn.benibix.com/term/ai/transformer)
- [FlashAttention](https://learn.benibix.com/term/ai/flash-attention)
- [KV Cache](https://learn.benibix.com/term/ai/kv-cache)

## Taught in

- [Self-Attention and Multi-Head](https://learn.benibix.com/learn/ai-llms/self-attention-and-multi-head): LLMs, 6/15

![One sentence feeds three attention heads. The first head draws a line from 'she' to 'Lily', the second from 'gave' to 'book'. Their outputs join into one box below.](https://learn.benibix.com/diagrams/ai9-multi-head-attention.svg)

---

Source: https://learn.benibix.com/term/ai/multi-head-attention
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
