Append once. Read the growing past.
Step through decoding and see keys and values accumulate in separate banks for each layer.
Preparing the 3D view…
Keep keys and values
Each layer retains the keys and values of earlier tokens. The violet bank stores keys; the green bank stores values. Columns are token positions and rows are three illustrative layers.
Append the new token
Changing the token count highlights the newest column in amber. Earlier columns stay in place. Thin paths from the current query show that attention still reads every cached key and uses the corresponding values.
Account for growing memory
Each tile stands for a whole per-token tensor, not one scalar. Head and feature dimensions are collapsed. This is a conceptual dynamic cache with three layers, not a measured memory layout or latency benchmark.
The equation behind the motion
Kₜ = concat(Kₜ₋₁, kₜ); Vₜ = concat(Vₜ₋₁, vₜ)
Synthetic inputs with explicit geometric assumptions. Rotate the view and inspect the model above.
Read the primary source