Why include a synthetic key-value retrieval task alongs...

In the context of evaluating LLM performance across long context windows, researchers often observe a ‘U-shaped’ performance curve where information at the beginning and end of the context is retrieved more accurately than information in the middle. While multi-document QA tasks are useful, they are susceptible to the objection that middle chunks are inherently more difficult to process or reason about.

By introducing a synthetic key-value retrieval task, the evaluator removes the complexity of reasoning. If a U-shaped performance curve still appears in this synthetic task, it provides clear evidence that the performance drop is a result of the model’s positional bias rather than the difficulty of the underlying content.