All cold checksBrowse all
GPU kernelsExpertAlgorithm derivation9 min
Situation
You must explain why tiled attention can be exact even though each query tile sees the key and value blocks one at a time and the softmax denominator is not known in advance.
Derive the running state needed for online softmax and explain how an old output accumulator is corrected when a later tile contains a larger maximum. Then state what FlashAttention changes and does not change about asymptotic compute and memory traffic.
Keep going