Given a FLOP budget of 6 × 10^23, how many parameters should your model have?

To determine the optimal parameter count, we start from the compute formula C = 6ND. By defining the loss model L = E + A/N^α + B/D^β and substituting D = C/(6N), we can differentiate with respect to N and set the result to zero. Given that α ≈ β ≈ 0.5, the exponent collapses, and using the empirical constant D*/N* ≈ 20, we arrive at the constant 120. For a budget of 6 × 10^23 FLOPs, this yields N* ≈ 70B parameters, which would be trained on approximately 1.4T tokens.