All cold checksBrowse all
Model evaluationExpertCausal diagnosis8 min
Situation
A new pretraining mixture produces a large gain on a public benchmark but almost no gain on private task data. The benchmark questions and common solution writeups existed online before the crawl cutoff.
Design a contamination audit that distinguishes memorized benchmark artifacts from genuine capability. Include tests that still work when exact string matching finds nothing.
Keep going