A benchmark you care about shows model scores plateauin...

When model scores plateau despite improvements elsewhere, you must investigate three competing explanations: a genuine capability ceiling on the benchmark’s difficulty, a label noise floor where the model cannot improve beyond the inherent error rate of the data, or a distribution shift where the benchmark no longer aligns with the model’s new strengths. The senior approach is to perform a manual audit of 100 randomly sampled errors to differentiate between a capability ceiling and a noise floor before assuming the model has hit a hard performance limit.