Positional Encoding

Unmasked self-attention without positional signals is permutation-equivariant: rearranging input tokens rearranges the outputs in the same way. It does not supply an independent notion of token position. A causal mask also constrains ordering, so the statement assumes no fixed causal mask.

Positional methods make position available to the model. Sinusoidal and learned absolute encodings modify input representations, while rotary encoding changes query and key vectors so their dot products reflect relative position.

Sinusoidal encoding gives each position a unique "fingerprint": a vector of sine and cosine values at many different frequencies. Low-frequency dimensions change slowly across positions (coarse location); high-frequency dimensions change fast (fine location). Because the fingerprint is built from sinusoids, the model can also represent relative offsets: the encoding of position pos + k is a fixed linear function of the encoding at pos.

You simply add the positional vector to the token embedding before the first attention layer, so every token carries both "what I am" and "where I am."

The original sinusoidal encoding, for position $pos$ and dimension index $i$ in a model of width $d$:

Each pair of dimensions $(2i, 2i+1)$ is a sinusoid with wavelength $2\pi \cdot 10000^{2i/d}$, geometrically increasing from $2\pi$ to just below $10000 \cdot 2\pi$. Two useful properties: it is defined for any position (although this does not guarantee good predictions beyond the training length), and $PE(pos+k)$ is a linear transform of $PE(pos)$, letting attention learn to attend by relative distance.

RoPE takes a different route: instead of adding to embeddings, it rotates the query and key vectors by an angle proportional to their position. The resulting score depends on the query/key content and on the relative offset $m-n$ through the rotation. Its positional dependence is relative; the score is not a function of distance alone. Longer-context performance still requires evaluation and may need position scaling or further training.

Key takeaway: Unmasked attention is permutation-equivariant, so order has to come from somewhere else: added to the embeddings (sinusoidal, learned) or rotated into the Q/K dot product (RoPE). Being able to compute an encoding beyond the training length does not guarantee reliable extrapolation.

1. Conceptual: Why does a Transformer need positional encoding at all? (Unmasked self-attention is permutation-equivariant, without position information it can't distinguish token order.)
2. Implementation: Do you add or concatenate positional encodings, and why add? (Add: it keeps dimensionality fixed and lets each dimension carry both content and position; concatenation is another design choice but changes the input width and projections.)
3. Applied: Why is sinusoidal encoding able to handle sequences longer than those seen in training? (It's a fixed function defined for any position, so encodings are defined beyond the training length; model accuracy there is not guaranteed.)
4. Systems-level: What problem does RoPE solve over absolute learned positions? (It makes the positional part of the attention score depend on relative offset without a finite learned position table. Long-context quality is a separate empirical question.)
5. Failure modes: What's the limitation of learned absolute positional embeddings? (The table has a fixed allocated length; extending it requires additional handling. Even allocated positions that were not trained need evaluation.)

From memory: write the sinusoidal PE formula, explain why you add rather than concatenate, and state one advantage of sinusoidal/RoPE over learned absolute positions. Check against Stage 3.

Sources and further reading

• Vaswani et al., Attention Is All You Need, §3.5: sinusoidal positions.
• Su et al., RoFormer: relative positional dependence through rotations.