Your gold set is 90% attributable, 10% non-attributable...

Sampling exemplars to match the 90:10 production traffic distribution is a mistake because it teaches the judge the base rate rather than the specific features that distinguish attributable from non-attributable content. A better approach is to stratify the exemplar set, specifically including difficult boundary cases. This strategy prioritizes minority recall and ensures the judge is robust, accepting a minor trade-off in majority-class precision for significantly better performance on the minority class.