Why do modern LLMs avoid using dropout during pre-train...

Modern LLMs generally omit dropout during pre-training for several reasons. First, because pre-training involves only one pass over the data, the model does not overfit individual examples. Second, dropout introduces unnecessary gradient noise. Empirically, the train-to-validation gap remains negligible with or without dropout, but the absolute training loss is often worse when dropout is applied.