Cold checks
Start where the definition stops.
These are not recall prompts for terminology. Each one starts with a failure, regression, or ambiguous experiment and asks you to isolate the cause before reaching for a fix.
No account required.
Your written answer stays in the page.
Only your self-assessment is saved in this browser.
Hard ones first
The reward improves. The model gets worse.
During a preference-optimization run, training preference accuracy keeps rising. Blind human evaluation and factuality peak early, then fall. Responses also become longer.
Work the problem
Your DDP scale-up changed the experiment.
A run moves from 8 to 64 GPUs. Per-device batch size and gradient accumulation stay unchanged, the loss uses mean reduction, and the learning-rate schedule is still expressed in epochs. Throughput improves, but the loss curve and final quality move.
Work the problem
The benchmark jumped after a data refresh.
A new pretraining mixture produces a large gain on a public benchmark but almost no gain on private task data. The benchmark questions and common solution writeups existed online before the crawl cutoff.
Work the problem
More failure surfaces
FlashAttention never materializes the score matrix.
You must explain why tiled attention can be exact even though each query tile sees the key and value blocks one at a time and the softmax denominator is not known in advance.
A resumed run diverges on the very next step.
An interrupted mixed-precision training job reloads the model and optimizer successfully. The first resumed batch has a different loss from an uninterrupted control run, even on the same hardware.
The right document was retrieved. The answer is still wrong.
Offline retrieval recall says the supporting document is in the top five. Production answers remain wrong, and adding more retrieved chunks sometimes makes them worse.
One task improves while another quietly collapses.
Joint training improves a high-resource generation task while a low-resource classification task falls below its single-task baseline. Aggregate validation loss still looks healthy.
Median latency is flat. P99 explodes under mixed lengths.
A serving stack meets its median latency target at steady load. When short chats and long-context requests share the queue, P99 time-to-first-token and inter-token latency both spike.
Training hangs on one batch with no stack trace.
A multi-node DDP job runs for hours and then hangs. GPU utilization drops on some ranks while others remain inside a collective. Restarting with the same seed reproduces the hang near the same batch.