Design an evaluation protocol that catches calibration...

Maintain a calibration benchmark consisting of 500-1,000 factual multiple-choice questions with verified ground truth. After each RLHF checkpoint, measure the Expected Calibration Error (ECE) at temperature 1 and track the temperature-scaling coefficient needed to reach a target ECE. A rising coefficient signals growing overconfidence, even if post-scaling ECE appears acceptable, because the model is producing increasingly peaked distributions. Treat both raw ECE and the scaling coefficient as go/no-go signals, flagging any checkpoint requiring a scaling temperature above 1.5 for review.