Distributed training7 min

A multi-node DDP job runs for hours and then hangs. GPU utilization drops on some ranks while others remain inside a collective. Restarting with the same seed reproduces the hang near the same batch.

Give a rank-by-rank debugging sequence that distinguishes uneven inputs, divergent control flow, unused parameters, and mismatched collectives. What evidence lets you stop blaming NCCL itself?

Your reasoning

Keep the reasoning, then revisit it.

These are your self-comparisons, separate from model feedback and account mastery. Sharing the question never shares your answer.

Pre-sign-in browser archive: 0 records.

Keep going

A different failure surface.

Browse all

Post-training

The reward improves. The model gets worse.

Open check

Model evaluation

The benchmark jumped after a data refresh.

Open check

GPU kernels

FlashAttention never materializes the score matrix.

Open check

All cold checks

Distributed training7 min

A multi-node DDP job runs for hours and then hangs. GPU utilization drops on some ranks while others remain inside a collective. Restarting with the same seed reproduces the hang near the same batch.

Give a rank-by-rank debugging sequence that distinguishes uneven inputs, divergent control flow, unused parameters, and mismatched collectives. What evidence lets you stop blaming NCCL itself?

Your reasoning

Keep the reasoning, then revisit it.

These are your self-comparisons, separate from model feedback and account mastery. Sharing the question never shares your answer.

Pre-sign-in browser archive: 0 records.

Keep going

A different failure surface.

Browse all

Post-training

The reward improves. The model gets worse.

Open check

Model evaluation

The benchmark jumped after a data refresh.

Open check

GPU kernels

FlashAttention never materializes the score matrix.

Open check