Why does SFT improve benchmark performance even with ve...

Benchmarks such as MMLU often utilize multiple-choice or short-answer formats. Because the base model already contains the necessary knowledge, the primary challenge is outputting the answer in the correct format. SFT effectively teaches the model to emit the expected output structure, meaning the performance improvement observed is largely a formatting effect rather than a gain in underlying knowledge.