Skip to content
BENIBIX Learn

Positional Encoding

AI💬 LLMsTransformer Architecture

Also known as Position Embedding, Rotary Position Embedding, RoPE

It adds each token's place in the text, so word order is not lost.

Think of it like

Think of it like seat numbers on a bus. They tell you who sits where in the line.

Example

Lucy's model reads 'not bad, really good' and 'not good, really bad'. Positional encoding is what lets it tell the two apart.

Why it matters

Order carries meaning. Without position, a transformer could not tell who did what to whom.

Definition

Positional encoding adds where each token sits in the text. Attention on its own treats text like a bag of tokens. Without this, 'dog bites man' and 'man bites dog' would look the same. The first transformer added a pattern of waves to each token's vector. Most models now use a method called rotary position embedding. It turns each vector by an angle based on where it sits.
The same three words, but the place numbers are not the same. So the model can tell the two lines apart.

Common mistake

Thinking the model reads left to right like a person. Attention sees all tokens at once. The position signal is the only reason that order counts.

Key takeaways

  1. 1
    Adds position. Where each token sits.
  2. 2
    Keeps order. Dog bites man is not man bites dog.
  3. 3
    Waves or rotation. Two common methods.

Learn it properly