You want to fit a Chinchilla scaling law for your model...

The primary issue is that the model’s terminal loss is heavily influenced by the specific learning rate schedule applied during training. An early checkpoint in a long cosine run is still operating at a higher learning rate compared to a model that has fully annealed. Consequently, these checkpoints are not equivalent to models trained from scratch for shorter durations. Relying on them leads to a biased scaling curve. The recommended approach is to use independent runs or a WSD (Warmup-Stable-Decay) schedule with checkpoint branching.