A new pretraining mixture produces a large gain on a public benchmark but almost no gain on private task data. The benchmark questions and common solution writeups existed online before the crawl cutoff.

Design a contamination audit that distinguishes memorized benchmark artifacts from genuine capability. Include tests that still work when exact string matching finds nothing.