Your model's confidence scores drive a routing decision...

Calibration, specifically — not just accuracy. A model can rank cases correctly relative to each other and still have probability values that don’t mean what they claim to mean. If a routing decision is thresholded on that confidence score, an uncalibrated model silently sends the wrong things down the expensive path, or worse, clears things it should have escalated, without ever showing up as a classification error.

In practice: check calibration with a reliability diagram (predicted probability vs. observed frequency, binned), correct it with Platt scaling or isotonic regression if it’s off, and re-check after every retrain — calibration drifts independently of accuracy, so passing accuracy checks doesn’t mean the thresholds are still trustworthy.