Llama 3 8B: when one conversation outweighs the model.
One long conversation can need as much memory as the model. Cache size for a single Llama 3 8B sequence on log scales. Sharing key and value heads (8 instead of 32) cuts it by four; an FP8 cache halves it again. The dashed line is the 16 GB of bf16 weights.