Position becomes a rotation.
Turn three coordinate pairs at different frequencies and inspect how position changes their phase.
Preparing the 3D view…
Split into pairs
A six-dimensional vector is split into three two-dimensional pairs. Each ring shows one pair, initially (1, 0). The rings are stacked only to separate them visually; depth is the pair index, not an extra embedding coordinate.
Move through the sequence
At position m, pair i rotates by m × 10000^(−2i/6) radians. The first pair turns rapidly; later pairs change more slowly. Every pair keeps its length.
Compare relative positions
Applying these rotations to queries and keys makes their inner product depend on their relative position. This scene shows the rotations only; it does not compute attention, train embeddings, or model long-context extensions.
The equation behind the motion
θᵢ = 10000^(−2i/6); (x₂ᵢ, x₂ᵢ₊₁) ← R(mθᵢ)(x₂ᵢ, x₂ᵢ₊₁)
Synthetic inputs with explicit geometric assumptions. Rotate the view and inspect the model above.
Read the primary source