Why does end-to-end accuracy fall when increasing k in...

The residual stream has a fixed width regardless of the number of tokens in the context. As you increase k, the attention mechanism must distribute its softmax mass across more positions, which can lead to averaging when it cannot clearly select the correct token. This creates a ‘funnel’ effect where the signal is diluted. Patching the stream from a successful low-k run into the high-k run can confirm if the failure is in the assembly (the model’s processing) rather than the retrieval.