Your reranker returns 10 candidates. The relevance scor...

The cliff heuristic is a standard approach for optimizing context length. When a reranker shows a sharp decline in relevance scores, it indicates that candidates beyond that point have a low probability of containing the correct answer. Including these low-probability candidates in the generator’s context increases the risk of the model hallucinating or failing to answer correctly, often referred to as the below-closed-book penalty. Therefore, it is more efficient to truncate at the cliff rather than sending all candidates.