How do you compare two retrievers when the prompt forma...

When prompt formatting creates significant variance in accuracy, it acts as a confound that masks the true performance of the retrievers. To properly compare them, you must move beyond measuring the delta at a single shared prompt format. Instead, you should focus on the difference of per-arm maxima, which represents the optimal performance each retriever can achieve under its best conditions. This approach requires establishing a rigorous evaluation protocol, weighing the trade-offs between the complexity of the testing methodology and the accuracy of the resulting comparison.