Walk me through how you would run an isoFLOP experiment to estimate the compute-optimal model size for your team’s next training run.

To conduct an isoFLOP experiment, first select 5-7 compute budgets spanning 2-3 orders of magnitude below your target. At each budget, train 5-8 model sizes ranging from approximately 0.3x to 3x the Chinchilla-optimal size. Fit a quadratic to the loss-vs-model-size curve at each fixed compute point, extract the minimum, and plot the optimal parameter count (N) versus compute (C) on a log-log axis. Finally, fit a power law to these points to extrapolate the optimal size for your target compute. This pilot should be efficient, using less than 1% of the total budget.