What happens to model quality if you skip language filtering?

When language filtering is omitted, non-target language tokens consume a portion of the compute budget that would otherwise be used for the target language. This dilution results in noticeably lower benchmark scores, as demonstrated by models like BLOOM, where a lower percentage of English pre-training data resulted in weaker English performance compared to models trained on exclusively English corpora of similar size.