← All questions
RAGHard· Attribution· Context Window· Redundancy· Evaluation

Your leave-one-out attribution says no chunk in the con...

The leave-one-out attribution method fails in scenarios where the context contains redundant information. If two chunks carry the same fact, removing one chunk does not change the model’s output, leading the method to report a zero marginal contribution for both. This accurately reflects the math but misleads the developer regarding the importance of the information. A more robust approach is to use Shapley values, which can split credit equally (e.g., 0.5/0.5) between the redundant chunks. However, this method is significantly more expensive, requiring a much higher number of replays to calculate.

Code playground (Python)