We are doing continued pre-training to improve Hindi on...

Extending a vocabulary for continued pre-training is an expensive compromise. Adding new embedding rows requires random initialization, which necessitates different learning rate treatments compared to the pre-trained rows. Furthermore, the model’s internal probability distributions are tuned to the original 32k vocabulary, creating a mismatch with the new tokens. Significant continued pre-training is required to make these new tokens useful, making training from scratch with a properly designed vocabulary the preferred engineering approach.