Softmax and Cross-Entropy Numerics
Subtracting the max prevents overflow, not rounding. Where the target gradient silently falls to zero, and the rewrite that keeps it.
Jump to a section
A loss of exactly zero
Give a standard fp32 softmax cross-entropy the logits with target class 0. The loss comes back as exactly 0 and the target logit's gradient as exactly 0, while the other two gradients stay positive at about each. The true gradient of this loss always sums to zero across the logits. This one no longer does.
Every framework subtracts the maximum logit before exponentiating, so that exp cannot overflow. Barron, Gnanapragasam and Tschannen showed in October 2026 that this fix is necessary but not sufficient: the standard implementations in JAX, PyTorch and textbooks still lose the target gradient once the model is confident enough.
The cause is ordinary floating point. Adding a small enough number to 1.0 returns exactly 1.0. When the softmax denominator rounds to 1, the target probability is exactly 1, the loss is exactly 0, and the target gradient, computed as that probability minus 1, is exactly 0.
The logit margin and the precision cliff
Confidence is easiest to measure with the logit margin , how far the target logit sits above the log-sum-exp of the others:
For that is . The loss and the target gradient depend only on :
The two expressions are equal in exact arithmetic. As grows, the true gradient shrinks smoothly like , all the way to underflow. The standard computation forms first and subtracts 1 from its reciprocal. Once is no larger than half the spacing between floating-point numbers just above 1, the sum rounds to 1 and the gradient falls off a precision cliff.
That spacing fixes where the cliff sits. Above 1, fp32 numbers are apart, so rounds to 1 once , at . bf16 carries 8 significant bits, which puts its cliff at ; fp64 carries 53, for a cliff at . In probability terms the bf16 cliff is a target probability of , about 99.6%, which is an ordinary confident prediction.
Keep the target gradient past the precision cliff
Standard bf16 and fp32 gradients end abruptly; the fixed fp32 expression keeps the small numerator.
Exact inputs and equations appear with the figure.
Preparing the scientific figure…
Scalar margin arithmetic with fp32 rounding and emulated round-to-nearest-even bf16. Zero gradients are omitted on this logarithmic axis, so the standard curves end. This does not reproduce a particular fused kernel or mixed-precision training run.
Barron et al., A Fix for Softmax and Cross-EntropySection 2, Equation 3
Experiment with related Studio inputs
Follow each line to the right. The standard computation steps down, then stops where 1 + e^-m rounds to 1: about m = 5.5 in bf16 and 16.6 in fp32. The fixed computation keeps going.
Computed with real fp32 rounding and emulated bf16 rounding for a single target logit. Points where the gradient is exactly zero are omitted, which is why the standard lines end.
Inspect the related Studio values
x,y,series 0,-0.3010299956639812,Standard (bf16) 0.25,-0.359021942641668,Standard (bf16) 0.5,-0.4214682310456047,Standard (bf16) 0.75,-0.4944261129281329,Standard (bf16) 1,-0.5693908745745943,Standard (bf16) 1.25,-0.6523651096393581,Standard (bf16) 1.5,-0.7454821336302755,Standard (bf16) 1.75,-0.8284563686950394,Standard (bf16) 2,-0.9311187105921871,Standard (bf16) 2.25,-1.0102999566398119,Standard (bf16) 2.5,-1.1294863643590205,Standard (bf16) 2.75,-1.2321487062561682,Standard (bf16) 3,-1.3668472801536244,Standard (bf16) 3.25,-1.4082399653118496,Standard (bf16) 3.5,-1.505149978319906,Standard (bf16) 3.75,-1.630088714928206,Standard (bf16) 4,-1.806179973983887,Standard (bf16) 4.25,-1.806179973983887,Standard (bf16) 4.5,-2.1072099696478683,Standard (bf16) 4.75,-2.1072099696478683,Standard (bf16) 5,-2.1072099696478683,Standard (bf16) 5.25,-2.1072099696478683,Standard (bf16) 5.5,-2.1072099696478683,Standard (bf16) 0,-0.3010299956639812,Standard (fp32) 0.25,-0.3587009569476138,Standard (fp32) 0.5,-0.4230362831219311,Standard (fp32) 0.75,-0.49373682567775584,Standard (fp32) 1,-0.5703423347604135,Standard (fp32) 1.25,-0.6522795508640861,Standard (fp32) 1.5,-0.7389143153112003,Standard (fp32) 1.75,-0.8295996013365485,Standard (fp32) 2,-0.9237129184890965,Standard (fp32) 2.25,-1.0206816226150572,Standard (fp32) 2.5,-1.1199976054673955,Standard (fp32) 2.75,-1.2212218816936253,Standard (fp32) 3,-1.3239847289527833,Standard (fp32) 3.25,-1.4279777049693034,Standard (fp32) 3.5,-1.532950799536737,Standard (fp32) 3.75,-1.638700019870078,Standard (fp32) 4,-1.7450596640856364,Standard (fp32) 4.25,-1.8519035955138177,Standard (fp32) 4.5,-1.959123872847957,Standard (fp32) 4.75,-2.066637976304475,Standard (fp32) 5,-2.174390417101289,Standard (fp32) 5.25,-2.2823248728561216,Standard (fp32) 5.5,-2.390394582190755,Standard (fp32) 5.75,-2.4985715886686997,Standard (fp32) 6,-2.6068497394352015,Standard (fp32) 6.25,-2.715167927340134,Standard (fp32) 6.5,-2.8235606969117595,Standard (fp32) 6.75,-2.9319987581610056,Standard (fp32) 7,-3.0404828669191772,Standard (fp32) 7.25,-3.1489540177198143,Standard (fp32) 7.5,-3.257405977648465,Standard (fp32) 7.75,-3.3660022810752572,Standard (fp32) 8,-3.4745201681063667,Standard (fp32) 8.25,-3.5829484252815895,Standard (fp32) 8.5,-3.6915836076569097,Standard (fp32) 8.75,-3.8001649193288354,Standard (fp32) 9,-3.908749550478631,Standard (fp32) 9.25,-4.017354858466477,Standard (fp32) 9.5,-4.125730256534371,Standard (fp32) 9.75,-4.234381041147947,Standard (fp32) 10,-4.342764924595948,Standard (fp32) 10.25,-4.450933450954355,Standard (fp32) 10.5,-4.560077920379423,Standard (fp32) 10.75,-4.6684173951682615,Standard (fp32) 11,-4.77756186459333,Standard (fp32) 11.25,-4.886263402330944,Standard (fp32) 11.5,-4.994270974557275,Standard (fp32) 11.75,-5.104145964729699,Standard (fp32) 12,-5.2076865566367685,Standard (fp32) 12.25,-5.321629908943605,Standard (fp32) 12.5,-5.432328206437295,Standard (fp32) 12.75,-5.543478658559962,Standard (fp32) 13,-5.644936299318738,Standard (fp32) 13.25,-5.747598641215887,Standard (fp32) 13.5,-5.844508654223943,Standard (fp32) 13.75,-5.969447390832243,Standard (fp32) 14,-6.078591860257311,Standard (fp32) 14.25,-6.224719895935548,Standard (fp32) 14.5,-6.321629908943605,Standard (fp32) 14.75,-6.446568645551905,Standard (fp32) 15,-6.446568645551905,Standard (fp32) 15.25,-6.622659904607587,Standard (fp32) 15.5,-6.622659904607587,Standard (fp32) 15.75,-6.923689900271567,Standard (fp32) 16,-6.923689900271567,Standard (fp32) 16.25,-6.923689900271567,Standard (fp32) 16.5,-6.923689900271567,Standard (fp32) 0,-0.3010299956639812,Fixed (fp32) 0.25,-0.35870092738550835,Fixed (fp32) 0.5,-0.42303624883957314,Fixed (fp32) 0.75,-0.49373682567775584,Fixed (fp32) 1,-0.5703422866347501,Fixed (fp32) 1.25,-0.652279492745647,Fixed (fp32) 1.5,-0.7389144217353153,Fixed (fp32) 1.75,-0.8295998198982882,Fixed (fp32) 2,-0.9237131085030447,Fixed (fp32) 2.25,-1.0206817583576406,Fixed (fp32) 2.5,-1.1199976054673955,Fixed (fp32) 2.75,-1.2212220432475993,Fixed (fp32) 3,-1.3239846607253358,Fixed (fp32) 3.25,-1.4279782684327815,Fixed (fp32) 3.5,-1.5329511307034278,Fixed (fp32) 3.75,-1.6386996678120267,Fixed (fp32) 4,-1.7450603387157104,Fixed (fp32) 4.25,-1.8519026464367008,Fixed (fp32) 4.5,-1.9591231365765387,Fixed (fp32) 4.75,-2.0666400511039607,Fixed (fp32) 5,-2.1743888458435823,Fixed (fp32) 5.25,-2.2823190228901367,Fixed (fp32) 5.5,-2.390390855628095,Fixed (fp32) 5.75,-2.4985733415786435,Fixed (fp32) 6,-2.6068420511728503,Fixed (fp32) 6.25,-2.715178082304978,Fixed (fp32) 6.5,-2.823566590721971,Fixed (fp32) 6.75,-2.931995991399,Fixed (fp32) 7,-3.0404571998533285,Fixed (fp32) 7.25,-3.1489432956257986,Fixed (fp32) 7.5,-3.257448780005576,Fixed (fp32) 7.75,-3.3659692537994776,Fixed (fp32) 8,-3.474501510733053,Fixed (fp32) 8.25,-3.5830429254061245,Fixed (fp32) 8.5,-3.6915914674755683,Fixed (fp32) 8.75,-3.800145532982693,Fixed (fp32) 9,-3.908703914338644,Fixed (fp32) 9.25,-4.017265677038108,Fixed (fp32) 9.5,-4.125830092216643,Fixed (fp32) 9.75,-4.234396517527198,Fixed (fp32) 10,-4.34296454000104,Fixed (fp32) 10.25,-4.45153380379883,Fixed (fp32) 10.5,-4.5601040267709045,Fixed (fp32) 10.75,-4.668674996668983,Fixed (fp32) 11,-4.777246543035809,Fixed (fp32) 11.25,-4.88581857015582,Fixed (fp32) 11.5,-4.994390936346622,Fixed (fp32) 11.75,-5.102963586044208,Fixed (fp32) 12,-5.211536478114386,Fixed (fp32) 12.25,-5.32010948452557,Fixed (fp32) 12.5,-5.428682639654707,Fixed (fp32) 12.75,-5.537255908152309,Fixed (fp32) 13,-5.645829263530775,Fixed (fp32) 13.25,-5.754402668113415,Fixed (fp32) 13.5,-5.862976105450714,Fixed (fp32) 13.75,-5.971549578999756,Fixed (fp32) 14,-6.080123100761703,Fixed (fp32) 14.25,-6.188696616782098,Fixed (fp32) 14.5,-6.297270179030691,Fixed (fp32) 14.75,-6.4058437730501785,Fixed (fp32) 15,-6.514417392670421,Fixed (fp32) 15.25,-6.622990956971233,Fixed (fp32) 15.5,-6.73156456977471,Fixed (fp32) 15.75,-6.840138112528796,Fixed (fp32) 16,-6.9487117609513405,Fixed (fp32) 16.25,-7.057285359703564,Fixed (fp32) 16.5,-7.165858985625292,Fixed (fp32) 16.75,-7.274432580786749,Fixed (fp32) 17,-7.383006182284615,Fixed (fp32) 17.25,-7.491579806504776,Fixed (fp32) 17.5,-7.600153420331542,Fixed (fp32) 17.75,-7.708727044217298,Fixed (fp32) 18,-7.817300693906956,Fixed (fp32) 18.25,-7.925874297122894,Fixed (fp32) 18.5,-8.034447903262397,Fixed (fp32) 18.75,-8.143021546873614,Fixed (fp32) 19,-8.251595155272389,Fixed (fp32) 19.25,-8.360168798216145,Fixed (fp32) 19.5,-8.468742388191194,Fixed (fp32) 19.75,-8.5773160195568,Fixed (fp32) 20,-8.685889623372777,Fixed (fp32) 20.25,-8.794463256809427,Fixed (fp32) 20.5,-8.90303687772196,Fixed (fp32) 20.75,-9.011610480690402,Fixed (fp32) 21,-9.120184104931619,Fixed (fp32) 21.25,-9.228757738634,Fixed (fp32) 21.5,-9.337331368039683,Fixed (fp32) 21.75,-9.445904978546114,Fixed (fp32) 22,-9.554478600613402,Fixed (fp32) 22.25,-9.663052235979555,Fixed (fp32) 22.5,-9.77162585586622,Fixed (fp32) 22.75,-9.8801994706676,Fixed (fp32) 23,-9.988773087631898,Fixed (fp32) 23.25,-10.097346705407361,Fixed (fp32) 23.5,-10.205920311625682,Fixed (fp32) 23.75,-10.31449394692874,Fixed (fp32) 24,-10.42306758552304,Fixed (fp32)
The chart computes both versions with real fp32 and emulated bf16 rounding. Before the cliff the standard curve already moves in visible steps; past it, the line simply ends.
The non-target gradients never added 1 to their numerators. Dividing them by a denominator that rounded to 1 leaves them intact, and that asymmetry is what breaks the zero sum.
Compute from the differences
The fix computes the loss as from the differences , and sets . The gradients then sum to zero by construction. The target gradient keeps in its numerator, so dividing by a denominator that rounded to 1 still returns about .
Softplus has to be computed with the same care: for , and otherwise. Neither branch ever forms 1 plus a tiny number and then subtracts the 1.
The whole effect can be reproduced without a GPU. Python floats are fp64; packing one through struct rounds it to fp32, and rounding the fp32 bits to their top 16 gives bf16.
1import math
2import struct
3
4
5def f32(x: float) -> float:
6 """Round a Python float (fp64) to the nearest fp32 value."""
7 return struct.unpack("f", struct.pack("f", x))[0]
8
9
10def bf16(x: float) -> float:
11 """Round to the nearest bf16 value (round half to even on the fp32 bits)."""
12 bits = struct.unpack("I", struct.pack("f", x))[0]
13 bits = (bits + 0x7FFF + ((bits >> 16) & 1)) & 0xFFFF0000
14 return struct.unpack("f", struct.pack("I", bits))[0]
15
16
17def target_gradients(m: float, rnd=f32):
18 e = rnd(math.exp(-m))
19 denominator = rnd(1 + e)
20 standard = rnd(rnd(1 / denominator) - 1) # 1/(1+e^-m) - 1
21 fixed = rnd(-e / denominator) # -e^-m / (1+e^-m)
22 return standard, fixed, denominator
23
24
25# Logits [9, -9, -9] with target 0: m = 18 - log 2, about 17.3
26standard, fixed, denominator = target_gradients(18 - math.log(2))
27others = 2 * f32(math.exp(-18) / denominator) # the two non-target gradients
28assert denominator == 1.0 and standard == 0.0
29assert abs(fixed + 3.05e-8) < 1e-10
30assert standard + others > 0 # the zero sum is broken
31assert abs(fixed + others) < 1e-15 # the fixed gradient restores it
32
33
34def cliff(rnd, limit):
35 """First margin, in steps of 0.01, where 1 + e^-m rounds to exactly 1."""
36 return next(i / 100 for i in range(limit) if rnd(1 + rnd(math.exp(-i / 100))) == 1.0)
37
38
39assert cliff(bf16, 1000) == 5.55 # 8 ln 2 = 5.545
40assert cliff(f32, 3000) == 16.64 # 24 ln 2 = 16.636
41assert cliff(float, 6000) == 36.74 # 53 ln 2 = 36.737In code
A PyTorch version of the fixed hard-target loss for one batch. Autograd produces the correct gradient because the target slot enters only through the differences.
1import torch
2import torch.nn.functional as F
3
4def cross_entropy_fixed(logits: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
5 # logits: [B, K]; target: [B] class indices
6 z_t = logits.gather(1, target[:, None]) # [B, 1]
7 diffs = logits - z_t # z_j - z_t
8 diffs = diffs.scatter(1, target[:, None], float("-inf")) # drop the target slot
9 margin = -torch.logsumexp(diffs, dim=1) # m
10 return F.softplus(-margin).mean() # log(1 + e^-m)
11
12z = torch.tensor([[9.0, -9.0, -9.0]], requires_grad=True)
13t = torch.tensor([0])
14cross_entropy_fixed(z, t).backward()
15print(z.grad) # target entry is small and negative, not 0; row sums to 0In the paper's fp32 JAX example, the standard computation on this row returns a loss and target gradient of exactly 0, because its margin of about 17.3 is past the fp32 cliff. The fixed version returns a loss of about 3.05e-8 and a target gradient of about -3.05e-8. The PyTorch code above implements the same hard-target identity; exact rounding depends on the kernel and accumulation precision.
Why a broken zero sum becomes a loss spike
Adam divides each update by a running estimate of the gradient's magnitude. When every example in a batch sits past the cliff, the target gradients vanish and that estimate decays toward zero. When an example falls back below the cliff, its gradient returns and is divided by a tiny denominator, which produces a large step.
The paper shows that this mechanism explains the "Slingshot" loss spikes Thilak et al. reported during grokking, and that the fixed loss removes them. A rounding error worth in the loss becomes a visible event in the training curve because Adam's normaliser shrinks while the gradient is missing.
The same cancellation in distillation and attention
Distillation has a second version of the bug. With a soft target from the teacher and student probabilities , the gradient with respect to the student logits is . Computing and separately and subtracting two nearly equal probabilities loses precision, so the gradient can stay non-zero even when student and teacher logits are identical. The paper computes directly from in log space, with expm1 and log1p.
The softmax Jacobian always sums to zero across a row, and FlashAttention's backward pass relies on that. For one query row, with , and subtracting is what makes each row of sum to zero. Low-precision arithmetic can break that identity just as it breaks the cross-entropy one, so check it explicitly when testing an attention kernel.
Catching it in a training run
Check the zero sum. Per-example gradients with respect to the logits should sum to zero. A unit test that compares loss values with a tolerance will pass on this bug, since the missing loss is around in fp32; the zero-sum check fails immediately.
Count examples past the cliff. Log the fraction of training examples whose margin exceeds the cliff for the dtype the loss is computed in: 5.5 for bf16, 16.6 for fp32. A model that keeps fitting its training set pushes that fraction up over time.
Watch the dtype of the loss, not the model. The cliff is set by the precision in which is formed. A model trained in bf16 whose loss is computed in fp32 has the fp32 cliff; a loss computed in bf16 hits its cliff at a target probability of about 99.6%.
Sources and further reading
- Barron, Gnanapragasam & Tschannen, A Fix for Softmax and Cross-Entropy: Softmax Collapse, Softmax Cancellation, and the Slingshot mechanism.
- Thilak et al., The Slingshot Mechanism: the loss spikes observed during grokking.
- Dao et al., FlashAttention: the tiled forward pass and the rowsum term in the backward pass.
- Higham, Accuracy and Stability of Numerical Algorithms: rounding and stable summation.