Skip to content
Sign in
All cold checks
LLM servingExpertSystems diagnosis8 min

Situation

A serving stack meets its median latency target at steady load. When short chats and long-context requests share the queue, P99 time-to-first-token and inter-token latency both spike.

Separate queueing, prefill compute, decode bandwidth, batching policy, and KV-cache pressure with measurements. Then propose mitigations that target the diagnosed phase instead of merely adding GPUs.

Keep going

A different failure surface.

Post-training

The reward improves. The model gets worse.

Open check

Distributed training

Your DDP scale-up changed the experiment.

Open check

Model evaluation

The benchmark jumped after a data refresh.

Open check