💬Multi-Head Attention
Also known as Multi-Head Self-Attention, Attention Heads, Attention Head
Many attention steps run side by side. Each finds its own links.
Think of it like
It is like a team of readers. One looks for names, one for times, and one for places.
Example
In the line 'Sam gave Lily a book because she asked', one head links 'she' to 'Lily'. Another head links 'gave' to 'book'.
Why it matters
One head can only weigh the tokens one way. Many heads let a model see many kinds of links at once.
Definition
Common mistake
Key takeaways
- 1Many heads. Run side by side.
- 2Own weights. Each looks for its own links.
- 3Joined at the end. Mixed into one output.