Distributed trainingExpertIncident diagnosis7 min

Situation

A multi-node DDP job runs for hours and then hangs. GPU utilization drops on some ranks while others remain inside a collective. Restarting with the same seed reproduces the hang near the same batch.

Give a rank-by-rank debugging sequence that distinguishes uneven inputs, divergent control flow, unused parameters, and mismatched collectives. What evidence lets you stop blaming NCCL itself?

Your reasoning

Keep going

A different failure surface.

Browse all

Post-training

The reward improves. The model gets worse.

Open check

Model evaluation

The benchmark jumped after a data refresh.

Open check

GPU kernels

FlashAttention never materializes the score matrix.

Open check

All cold checks

Distributed trainingExpertIncident diagnosis7 min

Situation

A multi-node DDP job runs for hours and then hangs. GPU utilization drops on some ranks while others remain inside a collective. Restarting with the same seed reproduces the hang near the same batch.

Give a rank-by-rank debugging sequence that distinguishes uneven inputs, divergent control flow, unused parameters, and mismatched collectives. What evidence lets you stop blaming NCCL itself?

Your reasoning

Keep going

A different failure surface.

Browse all

Post-training

The reward improves. The model gets worse.

Open check

Model evaluation

The benchmark jumped after a data refresh.

Open check

GPU kernels

FlashAttention never materializes the score matrix.

Open check