All cold checksBrowse all
Distributed trainingExpertIncident diagnosis7 min
Situation
A multi-node DDP job runs for hours and then hangs. GPU utilization drops on some ranks while others remain inside a collective. Restarting with the same seed reproduces the hang near the same batch.
Give a rank-by-rank debugging sequence that distinguishes uneven inputs, divergent control flow, unused parameters, and mismatched collectives. What evidence lets you stop blaming NCCL itself?
Keep going