Distributed trainingExpertOptimization correctness8 min

Situation

A run moves from 8 to 64 GPUs. Per-device batch size and gradient accumulation stay unchanged, the loss uses mean reduction, and the learning-rate schedule is still expressed in epochs. Throughput improves, but the loss curve and final quality move.

Reconstruct the effective optimization step before and after the scale-up. Explain exactly what DDP averages, where accumulation changes the scale, and what must remain invariant to claim a mathematically comparable run.

Your reasoning

Keep going

A different failure surface.

Browse all

Post-training

The reward improves. The model gets worse.

Open check

Model evaluation

The benchmark jumped after a data refresh.

Open check

GPU kernels

FlashAttention never materializes the score matrix.

Open check