Attention as geometry

Dot products, a softmax and a weighted average, drawn as vectors.

Query direction Most attention: cat (62%)

Attention is a dot product, a softmax and a weighted average. The query q is compared with every key k; keys that point the same way score highest. Softmax turns the scores into weights, and the output is the weighted mix of the values v. Two dimensions instead of 64 or 128, the same arithmetic.