💬Self-Attention
Also known as Intra-Attention, Scaled Dot-Product Attention
Each word in a text looks at the other words in that same text.
Think of it like
Think of it like each word in a line. It looks at all the other words in that same line.
Example
Take 'Lucy gave Jack her book'. Self-attention lets 'her' check itself against 'Lucy' and 'Jack'. It leans on 'Lucy'.
Why it matters
It is the step that runs in every layer of a transformer. So it is where most of the work gets done.
Definition
Common mistake
Key takeaways
- 1Looks at itself. The same sequence.
- 2Query, key, value. Three vectors per token.
- 3Only looks back. When writing text.