What goes wrong when you try to extend a RoPE model to...

When a model using RoPE is applied to context lengths beyond its training distribution, the high-frequency dimensions of the rotation matrix wrap around, leading to a loss of positional signal and noisy attention scores. YaRN (Yet another RoPE extension with Positional Interpolation) solves this by interpolating the rotation frequencies. By applying an interpolation factor s = L’/L, YaRN ensures that no dimension exceeds its trained range. The NTK-aware variant of this approach is generally preferred over naive linear interpolation, as it better preserves the performance of the model across the extended context window.