Our measured training FLOPs are 30% above the 6ND estim...

The 30% increase in FLOPs can be attributed to two specific factors. Activation checkpointing adds an extra forward pass, which accounts for exactly 33% of the base compute cost and remains constant regardless of context length. The second factor is the attention mechanism, which adds an s/12d term that scales with context length. By checking the context length and the checkpointing flag, one can isolate the cause; if the 30% overhead remains flat despite changes in context length, it is confirmed to be activation checkpointing.