How do you adjudicate a situation where a reranker has...

The classifier is being graded on a decision boundary, whereas the ranking metric is concerned with the relative order within the positive class. To diagnose this, one should measure nDCG restricted to queries with two or more relevant passages and calculate the fraction of errors where the gold document is ranked lower than a document the model also labeled as true. The fix is to move to a listwise loss, which requires more training effort but does not increase serving costs.