Was the use of dropout in GPT-3 a mistake?

Labeling GPT-3’s use of dropout as a mistake is incorrect. At the time of its training, the empirical consensus that pre-training overfitting was a non-issue at 175B parameters had not been established. Furthermore, the team used dropout partly as a hedge for training stability. This highlights that best practices in LLM training are empirically derived and evolve as model scale increases; what was considered reasonable caution in 2020 became unnecessary overhead by 2022.