💬Attention Mechanism
Also known as Attention, Attention Layer
How a model picks which other words matter for each word.
Think of it like
Think of it like a highlighter. For each word, you mark the other words that help explain it.
Example
Jack's model translates 'The bank by the river flooded'. Attention links 'bank' to 'river', so the model picks the word for a river bank, not a money bank.
Why it matters
Attention is why language models can keep track of who 'it' is in a long text.
Definition
Common mistake
Key takeaways
- 1Weights per token. Which others matter.
- 2Mixes in meaning. By those weights.
- 3Many heads, many layers. Run side by side.