Embeddings

An embedding represents a discrete item as a vector. A token embedding layer performs a table lookup; a sentence or document encoder computes a representation from a larger input.

The geometry is learned for a particular objective. Cosine similarity compares directions, while a dot product also depends on magnitude. Neither score by itself proves that two items mean the same thing or answer the same question.

Picture words as points in space. Training pulls words that share contexts together and pushes unrelated ones apart, until directions in the space become meaningful: the famous king − man + woman ≈ queen. Similarity is read off with a dot product or cosine.

In a Transformer, this lookup is the very first layer; the vectors are then refined by attention so the same token ends up with a context-dependent representation deeper in the network.

An embedding layer is a matrix $E \in \mathbb{R}^{V \times d}$ ($V$ = vocabulary size, $d$ = embedding dimension). Token id $t$ maps to row $E_t$: a lookup, equivalent to multiplying a one-hot vector by $E$.

word2vec (skip-gram) learns $E$ by predicting context words from a center word. For center word $c$ and context word $o$, it maximizes:

where $v_c$ is the center embedding and $u_o$ the context embedding. The full softmax over $V$ is expensive, so practical training uses negative sampling: distinguish the true context word from a few random "negative" words instead of normalizing over the whole vocabulary.

Similarity between two embeddings uses cosine, which ignores magnitude and compares direction:

Key takeaway: An embedding table is a learned $V \times d$ lookup where geometry carries meaning. Static embeddings freeze one vector per word; contextual embeddings recompute it per occurrence: that difference is what kills the "bank" ambiguity.

1. Conceptual: What's the difference between static and contextual embeddings? (Static, e.g. word2vec, gives one fixed vector per word; contextual, e.g. Transformer hidden states, gives a different vector per occurrence based on surrounding context.)
2. Implementation: Why use cosine similarity rather than raw dot product or Euclidean distance? (Cosine compares direction independent of magnitude, which suits embeddings where length can vary with token frequency.)
3. Applied: Why does word2vec use negative sampling? (The full softmax over the vocabulary is too expensive; negative sampling approximates it by separating the true context word from a few random negatives.)
4. Systems-level: How do embeddings enable semantic search and RAG? (Embed documents and the query into the same space; retrieve by nearest-neighbor similarity rather than keyword match.)
5. Failure modes: What does a single static embedding fail to capture, with an example? (Polysemy: "bank" (river) vs. "bank" (money) share one vector; contextual embeddings fix this.)

From memory: define an embedding layer and its shape, write the cosine-similarity formula, and explain static vs. contextual embeddings with one example. Check against Stage 3.