← All questions
EvaluationMedium· Sensitivity· Log-Likelihood· Decision Making

Template A scores 0.024 nats per token and template B scores 0.019. Ship B?

Sensitivity is evidence about accuracy, not a direct measurement of it. A lower sensitivity score (like 0.019 vs 0.024) suggests better stability, but it is a correlation. You must verify that the test conditions were identical and perform a labeled spot-check on a slice of the data to confirm that the lower sensitivity actually corresponds to higher accuracy.

Code playground (Python)