All cold checksBrowse all
Training engineeringAdvancedReproducibility7 min
Situation
An interrupted mixed-precision training job reloads the model and optimizer successfully. The first resumed batch has a different loss from an uninterrupted control run, even on the same hardware.
Give a fail-closed resume protocol. List the state that must be restored, then describe the shortest comparison that localizes whether the divergence begins in data, forward computation, backward computation, or the optimizer step.
Keep going