Why did cutting the reranker's candidate pool fail to significantly improve p50 latency?

Latency optimization must be guided by the distribution of compute costs. If the reranker accounts for only a few milliseconds of a multi-second budget, reducing its candidate pool will have no meaningful impact on the total p50. Furthermore, this change risks degrading quality by reducing recall. To improve p50, one must target the dominant cost, which is typically the decode phase, by adjusting output length, quantization, or batching policies.