Skip to content
BENIBIX Learn

Multi-Head Attention

AI💬 LLMsTransformer Architecture

Also known as Multi-Head Self-Attention, Attention Heads, Attention Head

Many attention steps run side by side. Each finds its own links.

Think of it like

It is like a team of readers. One looks for names, one for times, and one for places.

Example

In the line 'Sam gave Lily a book because she asked', one head links 'she' to 'Lily'. Another head links 'gave' to 'book'.

Why it matters

One head can only weigh the tokens one way. Many heads let a model see many kinds of links at once.

Definition

Multi-head attention runs several attention steps side by side on the same text. Each one, called a head, has its own learned weights, so it can notice a different kind of link between tokens. One head may track which name a word like 'she' points to, and another may track word order. The outputs of all the heads are joined and mixed into one result. The first transformer, in 2017, used 8 heads in each attention layer.
Each head finds a different link in the same sentence. Their outputs are joined into one.

Common mistake

Thinking a person gives each head its job. The jobs are learned in training, and many heads are hard to explain.

Key takeaways

  1. 1
    Many heads. Run side by side.
  2. 2
    Own weights. Each looks for its own links.
  3. 3
    Joined at the end. Mixed into one output.

Learn it properly