A vendor claims their new model has a 128k context wind...

Vendor-provided aggregate accuracy numbers are often misleading because they mask position-specific performance drops. To verify the window, you should run a synthetic key-value probe at various scales up to 128k tokens. By plotting the depth of the accuracy trough, you can determine if the model actually maintains performance across the entire context.

Furthermore, you must determine the method of extension. A model that underwent continued training at the longer length is significantly more likely to hold up in production than one that relies on a positional-encoding trick, which often fails to generalize to real-world tasks.