Why does a Transformer need positional encoding?

Self-attention is permutation-invariant (it has no built-in sense of token order) so positional encoding injects where each token sits in the sequence.

Do you add or concatenate positional encodings?

Add them to the token embeddings. Adding keeps the dimensionality fixed and lets each dimension carry both content and position; concatenation would waste model width.

What is RoPE (rotary positional encoding)?

RoPE rotates query and key vectors by a position-dependent angle so the attention dot product depends only on relative position, which generalizes better to long contexts than absolute encodings.

Why can sinusoidal encoding handle longer sequences than seen in training?

It's a fixed function defined for any position, so it produces valid encodings beyond the training length, unlike learned absolute embeddings.

Deep Learning

Positional Encoding

Compare positional signals at different frequencies, then connect absolute encodings and RoPE to attention.

7 min readReviewed September 2026By Oliver Perrin · Machine Learning Engineer

Learn with a guided session Read the full deep dive

Jump to a section

Why this matters

Unmasked self-attention without positional signals is permutation-equivariant: rearranging input tokens rearranges the outputs in the same way. It does not supply an independent notion of token position. A causal mask also constrains ordering, so the statement assumes no fixed causal mask.

Positional methods make position available to the model. Sinusoidal and learned absolute encodings modify input representations, while rotary encoding changes query and key vectors so their dot products reflect relative position.

Build the mental model

Sinusoidal encoding gives each position a unique "fingerprint": a vector of sine and cosine values at many different frequencies. Low-frequency dimensions change slowly across positions (coarse location); high-frequency dimensions change fast (fine location). Because the fingerprint is built from sinusoids, the model can also represent relative offsets: the encoding of position pos + k is a fixed linear function of the encoding at pos.

Explore the ideaOpen in Studio

Different frequencies encode different scales

Follow the fast and slow oscillations across positions. A positional vector combines multiple such coordinates.

Horizontal: Token positionVertical: Encoding coordinate

The visual loads as you reach this section.

Three exact coordinates from the classic sinusoidal encoding with d = 8. This is not a visualization of RoPE or learned positions.

Inspect the plotted values

x,y,series
0,0,sin(position)
0,0,sin(position / 10)
0,1,cos(position / 10)
1,0.8414709848078965,sin(position)
1,0.09983341664682815,sin(position / 10)
1,0.9950041652780257,cos(position / 10)
2,0.9092974268256817,sin(position)
2,0.19866933079506122,sin(position / 10)
2,0.9800665778412416,cos(position / 10)
3,0.1411200080598672,sin(position)
3,0.29552020666133955,sin(position / 10)
3,0.955336489125606,cos(position / 10)
4,-0.7568024953079282,sin(position)
4,0.3894183423086505,sin(position / 10)
4,0.9210609940028851,cos(position / 10)
5,-0.9589242746631385,sin(position)
5,0.479425538604203,sin(position / 10)
5,0.8775825618903728,cos(position / 10)
6,-0.27941549819892586,sin(position)
6,0.5646424733950354,sin(position / 10)
6,0.8253356149096783,cos(position / 10)
7,0.6569865987187891,sin(position)
7,0.644217687237691,sin(position / 10)
7,0.7648421872844885,cos(position / 10)
8,0.9893582466233818,sin(position)
8,0.7173560908995228,sin(position / 10)
8,0.6967067093471654,cos(position / 10)
9,0.4121184852417566,sin(position)
9,0.7833269096274834,sin(position / 10)
9,0.6216099682706644,cos(position / 10)
10,-0.5440211108893698,sin(position)
10,0.8414709848078965,sin(position / 10)
10,0.5403023058681398,cos(position / 10)
11,-0.9999902065507035,sin(position)
11,0.8912073600614354,sin(position / 10)
11,0.4535961214255773,cos(position / 10)
12,-0.5365729180004349,sin(position)
12,0.9320390859672263,sin(position / 10)
12,0.3623577544766736,cos(position / 10)
13,0.4201670368266409,sin(position)
13,0.963558185417193,sin(position / 10)
13,0.26749882862458735,cos(position / 10)
14,0.9906073556948704,sin(position)
14,0.9854497299884601,sin(position / 10)
14,0.16996714290024104,cos(position / 10)
15,0.6502878401571168,sin(position)
15,0.9974949866040544,sin(position / 10)
15,0.0707372016677029,cos(position / 10)
16,-0.2879033166650653,sin(position)
16,0.9995736030415051,sin(position / 10)
16,-0.029199522301288815,cos(position / 10)
17,-0.9613974918795568,sin(position)
17,0.9916648104524686,sin(position / 10)
17,-0.12884449429552464,cos(position / 10)
18,-0.7509872467716762,sin(position)
18,0.9738476308781951,sin(position / 10)
18,-0.2272020946930871,cos(position / 10)
19,0.14987720966295234,sin(position)
19,0.9463000876874145,sin(position / 10)
19,-0.32328956686350335,cos(position / 10)
20,0.9129452507276277,sin(position)
20,0.9092974268256817,sin(position / 10)
20,-0.4161468365471424,cos(position / 10)
21,0.8366556385360561,sin(position)
21,0.8632093666488737,sin(position / 10)
21,-0.5048461045998576,cos(position / 10)
22,-0.008851309290403876,sin(position)
22,0.8084964038195901,sin(position / 10)
22,-0.5885011172553458,cos(position / 10)
23,-0.8462204041751706,sin(position)
23,0.7457052121767203,sin(position / 10)
23,-0.666276021279824,cos(position / 10)
24,-0.9055783620066238,sin(position)
24,0.675463180551151,sin(position / 10)
24,-0.7373937155412454,cos(position / 10)
25,-0.13235175009777303,sin(position)
25,0.5984721441039564,sin(position / 10)
25,-0.8011436155469337,cos(position / 10)
26,0.7625584504796028,sin(position)
26,0.5155013718214642,sin(position / 10)
26,-0.8568887533689473,cos(position / 10)
27,0.956375928404503,sin(position)
27,0.4273798802338298,sin(position / 10)
27,-0.9040721420170612,cos(position / 10)
28,0.27090578830786904,sin(position)
28,0.3349881501559051,sin(position / 10)
28,-0.9422223406686581,cos(position / 10)
29,-0.6636338842129675,sin(position)
29,0.23924932921398243,sin(position / 10)
29,-0.9709581651495905,cos(position / 10)
30,-0.9880316240928618,sin(position)
30,0.1411200080598672,sin(position / 10)
30,-0.9899924966004454,cos(position / 10)
31,-0.404037645323065,sin(position)
31,0.04158066243329049,sin(position / 10)
31,-0.9991351502732795,cos(position / 10)
32,0.5514266812416906,sin(position)
32,-0.058374143427580086,sin(position / 10)
32,-0.9982947757947531,cos(position / 10)
33,0.9999118601072672,sin(position)
33,-0.1577456941432482,sin(position / 10)
33,-0.9874797699088649,cos(position / 10)
34,0.5290826861200238,sin(position)
34,-0.2555411020268312,sin(position / 10)
34,-0.9667981925794611,cos(position / 10)
35,-0.428182669496151,sin(position)
35,-0.35078322768961984,sin(position / 10)
35,-0.9364566872907963,cos(position / 10)
36,-0.9917788534431158,sin(position)
36,-0.44252044329485246,sin(position / 10)
36,-0.896758416334147,cos(position / 10)
37,-0.6435381333569995,sin(position)
37,-0.5298361409084934,sin(position / 10)
37,-0.848100031710408,cos(position / 10)
38,0.2963685787093853,sin(position)
38,-0.6118578909427189,sin(position / 10)
38,-0.7909677119144168,cos(position / 10)
39,0.9637953862840878,sin(position)
39,-0.6877661591839738,sin(position / 10)
39,-0.7259323042001402,cos(position / 10)
40,0.7451131604793488,sin(position)
40,-0.7568024953079282,sin(position / 10)
40,-0.6536436208636119,cos(position / 10)
41,-0.158622668804709,sin(position)
41,-0.8182771110644103,sin(position / 10)
41,-0.5748239465332692,cos(position / 10)
42,-0.9165215479156338,sin(position)
42,-0.8715757724135882,sin(position / 10)
42,-0.4902608213406994,cos(position / 10)
43,-0.8317747426285983,sin(position)
43,-0.9161659367494549,sin(position / 10)
43,-0.40079917207997545,cos(position / 10)
44,0.017701925105413577,sin(position)
44,-0.9516020738895161,sin(position / 10)
44,-0.30733286997841935,cos(position / 10)
45,0.8509035245341184,sin(position)
45,-0.977530117665097,sin(position / 10)
45,-0.2107957994307797,cos(position / 10)
46,0.9017883476488092,sin(position)
46,-0.9936910036334644,sin(position / 10)
46,-0.11215252693505487,cos(position / 10)
47,0.123573122745224,sin(position)
47,-0.9999232575641008,sin(position / 10)
47,-0.01238866346289056,cos(position / 10)
48,-0.7682546613236668,sin(position)
48,-0.9961646088358407,sin(position / 10)
48,0.0874989834394464,cos(position / 10)
49,-0.9537526527594719,sin(position)
49,-0.9824526126243325,sin(position / 10)
49,0.18651236942257576,cos(position / 10)
50,-0.26237485370392877,sin(position)
50,-0.9589242746631385,sin(position / 10)
50,0.28366218546322625,cos(position / 10)
51,0.6702291758433747,sin(position)
51,-0.9258146823277325,sin(position / 10)
51,0.37797774271298024,cos(position / 10)
52,0.9866275920404853,sin(position)
52,-0.8834546557201531,sin(position / 10)
52,0.4685166713003771,cos(position / 10)
53,0.39592515018183416,sin(position)
53,-0.8322674422239013,sin(position / 10)
53,0.5543743361791608,cos(position / 10)
54,-0.5587890488516163,sin(position)
54,-0.7727644875559871,sin(position / 10)
54,0.6346928759426347,cos(position / 10)
55,-0.9997551733586199,sin(position)
55,-0.7055403255703919,sin(position / 10)
55,0.70866977429126,cos(position / 10)
56,-0.5215510020869119,sin(position)
56,-0.6312666378723216,sin(position / 10)
56,0.7755658785102496,cos(position / 10)
57,0.43616475524782494,sin(position)
57,-0.5506855425976376,sin(position / 10)
57,0.8347127848391598,cos(position / 10)
58,0.9928726480845371,sin(position)
58,-0.46460217941375737,sin(position / 10)
58,0.8855195169413189,cos(position / 10)
59,0.6367380071391379,sin(position)
59,-0.373876664830236,sin(position / 10)
59,0.9274784307440359,cos(position / 10)
60,-0.3048106211022167,sin(position)
60,-0.27941549819892586,sin(position / 10)
60,0.9601702866503661,cos(position / 10)
61,-0.9661177700083929,sin(position)
61,-0.18216250427209588,sin(position / 10)
61,0.9832684384425845,cos(position / 10)
62,-0.7391806966492229,sin(position)
62,-0.0830894028174964,sin(position / 10)
62,0.9965420970232175,cos(position / 10)
63,0.16735570030280691,sin(position)
63,0.016813900484349713,sin(position / 10)
63,0.9998586363834151,cos(position / 10)

You simply add the positional vector to the token embedding before the first attention layer, so every token carries both "what I am" and "where I am."

Work through the math

The original sinusoidal encoding, for position $pos$ and dimension index $i$ in a model of width $d$ :

PE(pos, 2i) = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE(pos, 2i+1) = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)

Each pair of dimensions $(2i, 2i+1)$ is a sinusoid with wavelength $2\pi \cdot 10000^{2i/d}$ , geometrically increasing from $2\pi$ to $10000 \cdot 2\pi$ . Two useful properties: it's defined for any position (so it extrapolates to longer sequences than seen in training), and $PE(pos+k)$ is a linear transform of $PE(pos)$ , letting attention learn to attend by relative distance.

RoPE takes a different route: instead of adding to embeddings, it rotates the query and key vectors by an angle proportional to their position. The dot product $q_m \cdot k_n$ then depends only on the relative offset $m - n$ , baking relative position directly into attention, which generalizes better to long contexts.

Key takeaway: Unmasked attention is permutation-equivariant, so order has to come from somewhere else: added to the embeddings (sinusoidal, learned) or rotated into the Q/K dot product (RoPE). The trend in LLMs is toward relative schemes because they extrapolate.

Scheme	Type	Extrapolates past training length?	Used by
Sinusoidal	Absolute, fixed, parameter-free	Yes (defined for any position)	Original Transformer
Learned absolute	Absolute, trained table	No (undefined beyond max length)	BERT, GPT-2
ALiBi	Relative, linear attention bias	Yes (distance-based penalty)	BLOOM, MPT
RoPE	Relative, rotation of Q/K	Partially; extended via scaling tricks	Llama, Mistral, most modern LLMs

Read the implementation

python

1import torch
2from torch import Tensor
3
4
5def sinusoidal_encoding(seq_len: int, d_model: int) -> Tensor:
6    pos = torch.arange(seq_len).unsqueeze(1)                 # (seq, 1)
7    i = torch.arange(0, d_model, 2)                          # even indices
8    div = torch.exp(i * (-torch.log(torch.tensor(10000.0)) / d_model))
9    pe = torch.zeros(seq_len, d_model)
10    pe[:, 0::2] = torch.sin(pos * div)                       # even dims -> sin
11    pe[:, 1::2] = torch.cos(pos * div)                       # odd dims  -> cos
12    return pe
13
14
15pe = sinusoidal_encoding(seq_len=50, d_model=128)
16tokens = torch.randn(50, 128)
17position_aware = tokens + pe                                 # add, don't concat
18assert pe.shape == (50, 128)

Questions and trade-offs

Conceptual: Why does a Transformer need positional encoding at all? (Unmasked self-attention is permutation-equivariant, without position information it can't distinguish token order.)
Implementation: Do you add or concatenate positional encodings, and why add? (Add: it keeps dimensionality fixed and lets each dimension carry both content and position; concatenation wastes width.)
Applied: Why is sinusoidal encoding able to handle sequences longer than those seen in training? (It's a fixed function defined for any position, so it produces valid encodings beyond the training length.)
Systems-level: What problem does RoPE solve over absolute learned positions? (It encodes relative position directly in the attention dot product, generalizing better to long contexts than fixed absolute embeddings.)
Failure modes: What's the limitation of learned absolute positional embeddings? (They're only defined up to the trained max length and don't extrapolate: sequences longer than training have no embedding.)

Check your understanding

From memory: write the sinusoidal PE formula, explain why you add rather than concatenate, and state one advantage of sinusoidal/RoPE over learned absolute positions. Check against Stage 3.

Oliver Perrin

Machine Learning Engineer · Founder, LiminalML

LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.

Keep exploring the idea.

Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.

Ask a follow-up See pricing

Related concepts

Deep Learning

Transformer Architecture

Follow attention, feed-forward layers, normalization, and residual connections through a Transformer block.

Deep Learning

Attention Mechanisms

Turn compatibility scores into attention weights, then combine values. Explore the geometry, normalization, and limits of the mechanism.

LLMs

Embeddings

Compare vectors by direction and magnitude, and connect embedding geometry to retrieval and representation learning.

Deep Learning

Positional Encoding

Compare positional signals at different frequencies, then connect absolute encodings and RoPE to attention.

7 min readReviewed September 2026By Oliver Perrin · Machine Learning Engineer

Learn with a guided session Read the full deep dive

Jump to a section

Why this matters

Build the mental model

Explore the ideaOpen in Studio

Different frequencies encode different scales

Follow the fast and slow oscillations across positions. A positional vector combines multiple such coordinates.

Horizontal: Token positionVertical: Encoding coordinate

The visual loads as you reach this section.

Three exact coordinates from the classic sinusoidal encoding with d = 8. This is not a visualization of RoPE or learned positions.

Inspect the plotted values

x,y,series
0,0,sin(position)
0,0,sin(position / 10)
0,1,cos(position / 10)
1,0.8414709848078965,sin(position)
1,0.09983341664682815,sin(position / 10)
1,0.9950041652780257,cos(position / 10)
2,0.9092974268256817,sin(position)
2,0.19866933079506122,sin(position / 10)
2,0.9800665778412416,cos(position / 10)
3,0.1411200080598672,sin(position)
3,0.29552020666133955,sin(position / 10)
3,0.955336489125606,cos(position / 10)
4,-0.7568024953079282,sin(position)
4,0.3894183423086505,sin(position / 10)
4,0.9210609940028851,cos(position / 10)
5,-0.9589242746631385,sin(position)
5,0.479425538604203,sin(position / 10)
5,0.8775825618903728,cos(position / 10)
6,-0.27941549819892586,sin(position)
6,0.5646424733950354,sin(position / 10)
6,0.8253356149096783,cos(position / 10)
7,0.6569865987187891,sin(position)
7,0.644217687237691,sin(position / 10)
7,0.7648421872844885,cos(position / 10)
8,0.9893582466233818,sin(position)
8,0.7173560908995228,sin(position / 10)
8,0.6967067093471654,cos(position / 10)
9,0.4121184852417566,sin(position)
9,0.7833269096274834,sin(position / 10)
9,0.6216099682706644,cos(position / 10)
10,-0.5440211108893698,sin(position)
10,0.8414709848078965,sin(position / 10)
10,0.5403023058681398,cos(position / 10)
11,-0.9999902065507035,sin(position)
11,0.8912073600614354,sin(position / 10)
11,0.4535961214255773,cos(position / 10)
12,-0.5365729180004349,sin(position)
12,0.9320390859672263,sin(position / 10)
12,0.3623577544766736,cos(position / 10)
13,0.4201670368266409,sin(position)
13,0.963558185417193,sin(position / 10)
13,0.26749882862458735,cos(position / 10)
14,0.9906073556948704,sin(position)
14,0.9854497299884601,sin(position / 10)
14,0.16996714290024104,cos(position / 10)
15,0.6502878401571168,sin(position)
15,0.9974949866040544,sin(position / 10)
15,0.0707372016677029,cos(position / 10)
16,-0.2879033166650653,sin(position)
16,0.9995736030415051,sin(position / 10)
16,-0.029199522301288815,cos(position / 10)
17,-0.9613974918795568,sin(position)
17,0.9916648104524686,sin(position / 10)
17,-0.12884449429552464,cos(position / 10)
18,-0.7509872467716762,sin(position)
18,0.9738476308781951,sin(position / 10)
18,-0.2272020946930871,cos(position / 10)
19,0.14987720966295234,sin(position)
19,0.9463000876874145,sin(position / 10)
19,-0.32328956686350335,cos(position / 10)
20,0.9129452507276277,sin(position)
20,0.9092974268256817,sin(position / 10)
20,-0.4161468365471424,cos(position / 10)
21,0.8366556385360561,sin(position)
21,0.8632093666488737,sin(position / 10)
21,-0.5048461045998576,cos(position / 10)
22,-0.008851309290403876,sin(position)
22,0.8084964038195901,sin(position / 10)
22,-0.5885011172553458,cos(position / 10)
23,-0.8462204041751706,sin(position)
23,0.7457052121767203,sin(position / 10)
23,-0.666276021279824,cos(position / 10)
24,-0.9055783620066238,sin(position)
24,0.675463180551151,sin(position / 10)
24,-0.7373937155412454,cos(position / 10)
25,-0.13235175009777303,sin(position)
25,0.5984721441039564,sin(position / 10)
25,-0.8011436155469337,cos(position / 10)
26,0.7625584504796028,sin(position)
26,0.5155013718214642,sin(position / 10)
26,-0.8568887533689473,cos(position / 10)
27,0.956375928404503,sin(position)
27,0.4273798802338298,sin(position / 10)
27,-0.9040721420170612,cos(position / 10)
28,0.27090578830786904,sin(position)
28,0.3349881501559051,sin(position / 10)
28,-0.9422223406686581,cos(position / 10)
29,-0.6636338842129675,sin(position)
29,0.23924932921398243,sin(position / 10)
29,-0.9709581651495905,cos(position / 10)
30,-0.9880316240928618,sin(position)
30,0.1411200080598672,sin(position / 10)
30,-0.9899924966004454,cos(position / 10)
31,-0.404037645323065,sin(position)
31,0.04158066243329049,sin(position / 10)
31,-0.9991351502732795,cos(position / 10)
32,0.5514266812416906,sin(position)
32,-0.058374143427580086,sin(position / 10)
32,-0.9982947757947531,cos(position / 10)
33,0.9999118601072672,sin(position)
33,-0.1577456941432482,sin(position / 10)
33,-0.9874797699088649,cos(position / 10)
34,0.5290826861200238,sin(position)
34,-0.2555411020268312,sin(position / 10)
34,-0.9667981925794611,cos(position / 10)
35,-0.428182669496151,sin(position)
35,-0.35078322768961984,sin(position / 10)
35,-0.9364566872907963,cos(position / 10)
36,-0.9917788534431158,sin(position)
36,-0.44252044329485246,sin(position / 10)
36,-0.896758416334147,cos(position / 10)
37,-0.6435381333569995,sin(position)
37,-0.5298361409084934,sin(position / 10)
37,-0.848100031710408,cos(position / 10)
38,0.2963685787093853,sin(position)
38,-0.6118578909427189,sin(position / 10)
38,-0.7909677119144168,cos(position / 10)
39,0.9637953862840878,sin(position)
39,-0.6877661591839738,sin(position / 10)
39,-0.7259323042001402,cos(position / 10)
40,0.7451131604793488,sin(position)
40,-0.7568024953079282,sin(position / 10)
40,-0.6536436208636119,cos(position / 10)
41,-0.158622668804709,sin(position)
41,-0.8182771110644103,sin(position / 10)
41,-0.5748239465332692,cos(position / 10)
42,-0.9165215479156338,sin(position)
42,-0.8715757724135882,sin(position / 10)
42,-0.4902608213406994,cos(position / 10)
43,-0.8317747426285983,sin(position)
43,-0.9161659367494549,sin(position / 10)
43,-0.40079917207997545,cos(position / 10)
44,0.017701925105413577,sin(position)
44,-0.9516020738895161,sin(position / 10)
44,-0.30733286997841935,cos(position / 10)
45,0.8509035245341184,sin(position)
45,-0.977530117665097,sin(position / 10)
45,-0.2107957994307797,cos(position / 10)
46,0.9017883476488092,sin(position)
46,-0.9936910036334644,sin(position / 10)
46,-0.11215252693505487,cos(position / 10)
47,0.123573122745224,sin(position)
47,-0.9999232575641008,sin(position / 10)
47,-0.01238866346289056,cos(position / 10)
48,-0.7682546613236668,sin(position)
48,-0.9961646088358407,sin(position / 10)
48,0.0874989834394464,cos(position / 10)
49,-0.9537526527594719,sin(position)
49,-0.9824526126243325,sin(position / 10)
49,0.18651236942257576,cos(position / 10)
50,-0.26237485370392877,sin(position)
50,-0.9589242746631385,sin(position / 10)
50,0.28366218546322625,cos(position / 10)
51,0.6702291758433747,sin(position)
51,-0.9258146823277325,sin(position / 10)
51,0.37797774271298024,cos(position / 10)
52,0.9866275920404853,sin(position)
52,-0.8834546557201531,sin(position / 10)
52,0.4685166713003771,cos(position / 10)
53,0.39592515018183416,sin(position)
53,-0.8322674422239013,sin(position / 10)
53,0.5543743361791608,cos(position / 10)
54,-0.5587890488516163,sin(position)
54,-0.7727644875559871,sin(position / 10)
54,0.6346928759426347,cos(position / 10)
55,-0.9997551733586199,sin(position)
55,-0.7055403255703919,sin(position / 10)
55,0.70866977429126,cos(position / 10)
56,-0.5215510020869119,sin(position)
56,-0.6312666378723216,sin(position / 10)
56,0.7755658785102496,cos(position / 10)
57,0.43616475524782494,sin(position)
57,-0.5506855425976376,sin(position / 10)
57,0.8347127848391598,cos(position / 10)
58,0.9928726480845371,sin(position)
58,-0.46460217941375737,sin(position / 10)
58,0.8855195169413189,cos(position / 10)
59,0.6367380071391379,sin(position)
59,-0.373876664830236,sin(position / 10)
59,0.9274784307440359,cos(position / 10)
60,-0.3048106211022167,sin(position)
60,-0.27941549819892586,sin(position / 10)
60,0.9601702866503661,cos(position / 10)
61,-0.9661177700083929,sin(position)
61,-0.18216250427209588,sin(position / 10)
61,0.9832684384425845,cos(position / 10)
62,-0.7391806966492229,sin(position)
62,-0.0830894028174964,sin(position / 10)
62,0.9965420970232175,cos(position / 10)
63,0.16735570030280691,sin(position)
63,0.016813900484349713,sin(position / 10)
63,0.9998586363834151,cos(position / 10)

You simply add the positional vector to the token embedding before the first attention layer, so every token carries both "what I am" and "where I am."

Work through the math

The original sinusoidal encoding, for position $pos$ and dimension index $i$ in a model of width $d$ :

PE(pos, 2i) = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE(pos, 2i+1) = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)

Key takeaway: Unmasked attention is permutation-equivariant, so order has to come from somewhere else: added to the embeddings (sinusoidal, learned) or rotated into the Q/K dot product (RoPE). The trend in LLMs is toward relative schemes because they extrapolate.

Scheme	Type	Extrapolates past training length?	Used by
Sinusoidal	Absolute, fixed, parameter-free	Yes (defined for any position)	Original Transformer
Learned absolute	Absolute, trained table	No (undefined beyond max length)	BERT, GPT-2
ALiBi	Relative, linear attention bias	Yes (distance-based penalty)	BLOOM, MPT
RoPE	Relative, rotation of Q/K	Partially; extended via scaling tricks	Llama, Mistral, most modern LLMs

Read the implementation

python

1import torch
2from torch import Tensor
3
4
5def sinusoidal_encoding(seq_len: int, d_model: int) -> Tensor:
6    pos = torch.arange(seq_len).unsqueeze(1)                 # (seq, 1)
7    i = torch.arange(0, d_model, 2)                          # even indices
8    div = torch.exp(i * (-torch.log(torch.tensor(10000.0)) / d_model))
9    pe = torch.zeros(seq_len, d_model)
10    pe[:, 0::2] = torch.sin(pos * div)                       # even dims -> sin
11    pe[:, 1::2] = torch.cos(pos * div)                       # odd dims  -> cos
12    return pe
13
14
15pe = sinusoidal_encoding(seq_len=50, d_model=128)
16tokens = torch.randn(50, 128)
17position_aware = tokens + pe                                 # add, don't concat
18assert pe.shape == (50, 128)

Questions and trade-offs

Conceptual: Why does a Transformer need positional encoding at all? (Unmasked self-attention is permutation-equivariant, without position information it can't distinguish token order.)
Implementation: Do you add or concatenate positional encodings, and why add? (Add: it keeps dimensionality fixed and lets each dimension carry both content and position; concatenation wastes width.)
Applied: Why is sinusoidal encoding able to handle sequences longer than those seen in training? (It's a fixed function defined for any position, so it produces valid encodings beyond the training length.)
Systems-level: What problem does RoPE solve over absolute learned positions? (It encodes relative position directly in the attention dot product, generalizing better to long contexts than fixed absolute embeddings.)
Failure modes: What's the limitation of learned absolute positional embeddings? (They're only defined up to the trained max length and don't extrapolate: sequences longer than training have no embedding.)

Check your understanding

From memory: write the sinusoidal PE formula, explain why you add rather than concatenate, and state one advantage of sinusoidal/RoPE over learned absolute positions. Check against Stage 3.

Oliver Perrin

Machine Learning Engineer · Founder, LiminalML

LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.

Keep exploring the idea.

Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.

Ask a follow-up See pricing

Related concepts

Deep Learning

Transformer Architecture

Follow attention, feed-forward layers, normalization, and residual connections through a Transformer block.

Deep Learning

Attention Mechanisms

Turn compatibility scores into attention weights, then combine values. Explore the geometry, normalization, and limits of the mechanism.

LLMs

Embeddings

Compare vectors by direction and magnitude, and connect embedding geometry to retrieval and representation learning.