Our bi-encoder hits 99% accuracy on in-batch training,...

The discrepancy exists because the two tasks measure different things. The training task is saturated because the negatives are not challenging enough. The fix is to mine hard negatives from an ANN index over the current checkpoint and refresh them as the embedding space evolves. The frequency of refresh should be determined by the index rebuild cost.