Skip to content
BENIBIX Learn

Attention Mechanism

AI💬 LLMsTransformer Architecture

Also known as Attention, Attention Layer

How a model picks which other words matter for each word.

Think of it like

Think of it like a highlighter. For each word, you mark the other words that help explain it.

Example

Jack's model translates 'The bank by the river flooded'. Attention links 'bank' to 'river', so the model picks the word for a river bank, not a money bank.

Why it matters

Attention is why language models can keep track of who 'it' is in a long text.

Definition

The attention mechanism lets a model pick which other tokens matter most for each token. It gives each other token a weight, then mixes in their meaning by those weights. Take 'The animal did not cross the street because it was tired'. Attention helps the model link 'it' to 'animal'. A transformer runs many of these side by side in each layer.
The thicker the line from 'it', the more attention it pays to that word. Most goes to 'animal'.

Common mistake

Reading attention weights as the model's reasons. They show where information flowed, which is only part of why an answer came out.

Key takeaways

  1. 1
    Weights per token. Which others matter.
  2. 2
    Mixes in meaning. By those weights.
  3. 3
    Many heads, many layers. Run side by side.

Learn it properly