I double the number of training tokens. How does the va...

Validation loss follows a power-law relationship with training tokens. The loss drops by a factor of 2α, where α typically ranges from 0.05 to 0.30. For language models, α is approximately 0.1, meaning doubling the training tokens provides a roughly 7% relative improvement in loss, provided the model is large enough to remain in the scaling regime.