← All questions
EvaluationMedium· Benchmarking· Scaling Laws

Why not just train the new architecture at a small scal...

Small-scale benchmark testing is insufficient because architectures may rank differently on perplexity compared to specific benchmarks. Furthermore, emergent capabilities often appear discontinuously as models scale, meaning small-scale results do not necessarily predict large-scale performance. Perplexity scaling provides a more reliable signal, with benchmark validation reserved for after the architecture decision is finalized.

Code playground (Python)