Skip to content
BENIBIX Learn

Self-Attention

AI💬 LLMsTransformer Architecture

Also known as Intra-Attention, Scaled Dot-Product Attention

Each word in a text looks at the other words in that same text.

Think of it like

Think of it like each word in a line. It looks at all the other words in that same line.

Example

Take 'Lucy gave Jack her book'. Self-attention lets 'her' check itself against 'Lucy' and 'Jack'. It leans on 'Lucy'.

Why it matters

It is the step that runs in every layer of a transformer. So it is where most of the work gets done.

Definition

Self-attention is attention where a text looks at itself. Each token looks at the other tokens in the same text. Each token makes three vectors, called a query, a key and a value. The query of one token is checked against the key of each other token with a dot product. The scores decide how much of each value to mix in. In a model that writes text, each token may only look back, never ahead.
Every line here stays inside one sentence. That is what makes it self-attention.

Common mistake

Thinking the model can peek ahead while it writes. In a writing model, each token only sees the tokens before it.

Key takeaways

  1. 1
    Looks at itself. The same sequence.
  2. 2
    Query, key, value. Three vectors per token.
  3. 3
    Only looks back. When writing text.

Learn it properly