What is the WSD learning rate schedule and why did Mini...

The WSD schedule separates training into three distinct phases: a warmup phase, a flat stable phase, and a short, steep decay phase. The stable phase is highly reusable; any checkpoint taken during this phase can be branched into an independent decay phase to produce a terminal-quality model. This significantly reduces the compute cost of scaling analysis, as it replaces the need for K separate training runs with one long stable run plus K short decays.