If pre-training doesn't overfit, why is weight decay used?

Weight decay during pre-training serves a different purpose than in standard supervised learning. It does not aim to reduce the train-to-validation gap, but rather to improve the absolute loss level. By conditioning the parameter landscape, weight decay allows for more effective optimization during the learning rate cooldown phase, functioning as a tool for optimizer dynamics rather than regularization.