Decoding with a KV cache

Each step projects one key and value, then reads them all.

Step 1 of 8. Keys and values projected this step: 1, so far: 1

Each step projects one new key and value, then reads all of them. The newest query q scores every cached key k; the weights mix the cached values v into the output o. Switch off the cache to see the prefix projected again on every step. Weights are illustrative.