How does sliding-window attention affect long-range ret...

In a hybrid attention setup, local layers cannot attend to distant tokens, but global layers retain full context. Retrieval performance is generally maintained because global layers perform a full attention pass. The main risk is that if global layers are sparse, the ‘correct’ attention pattern might not emerge at the specific layer queried for a given output position.