What does linear attention lose versus softmax attentio...

Softmax attention is a sparse content router that can concentrate almost all probability mass on a single token, which is critical for precise tasks. Linear attention, by contrast, accumulates history into a fixed matrix via outer products. If two past keys map to similar feature directions, their values blend inseparably, which degrades performance on tasks requiring exact in-context lookup, multi-hop retrieval, or copy operations. However, this loss is often irrelevant for local coherence, syntax, or open-ended generation, allowing hybrid models with a mix of linear and full-attention layers to achieve near-transformer perplexity.