Gemma 2 applies a LayerNorm both before and after each...

Double-norm solves the issue where sub-blocks can produce large-magnitude outputs even when their inputs are normalized, which is particularly problematic at very large scales. By bounding the output before re-adding it to the residual stream, it acts as a second safety checkpoint. The cost of this approach involves extra memory bandwidth, as LayerNorm is bandwidth-bound rather than flop-bound, requiring an additional read/write pass over the activation tensor per layer, along with additional learned scale parameters.