179 questions
No questions match those filters.
← All questions
EvaluationMedium· Benchmarks· Training & Fine-tuning· Statistical Significance
A colleague says their new data mixture improved MMLU b...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansA 2-point improvement on MMLU after only 10B training tokens is generally not a reliable signal. At this scale, the variance of MMLU scores across different random seeds is typically larger than the reported 2-point gain. Instead of relying on MMLU, you should look at perplexity on domain-stratified validation slices. Ask the colleague to provide the perplexity curves before treating the MMLU improvement as a meaningful indicator of progress.