The gradient is p minus one-hot; naive float32 overflows.
Loss
Softmax turns logits into probabilities, the loss is −log of the target’s probability, and the gradient is p minus the one-hot target. Drag a logit bar or click a class to make it the target. Then add a constant to every logit: the probabilities never change, but exponentiating directly in float32 overflows and returns NaN.