💬Positional Encoding
Also known as Position Embedding, Rotary Position Embedding, RoPE
It adds each token's place in the text, so word order is not lost.
Think of it like
Think of it like seat numbers on a bus. They tell you who sits where in the line.
Example
Lucy's model reads 'not bad, really good' and 'not good, really bad'. Positional encoding is what lets it tell the two apart.
Why it matters
Definition
Common mistake
Key takeaways
- 1Adds position. Where each token sits.
- 2Keeps order. Dog bites man is not man bites dog.
- 3Waves or rotation. Two common methods.