Relative encodings claim to generalize to unseen length...

Relative encodings often fail to generalize perfectly. T5 clips relative distances at a maximum offset k, meaning the model cannot distinguish between positions beyond that distance. ALiBi avoids clipping but uses a monotone penalty that assumes recency is always the most important factor, which is inaccurate for many long-context retrieval tasks. True unlimited generalization requires the encoding to remain distinguishable at arbitrary distances, as seen in RoPE.