What activation function do frontier models use in the...

The dominant activation function in modern frontier models is SwiGLU, defined as SwiGLU(x, W, V, W2) = (xW * σ(xV ))W2. This gated variant of SiLU uses a gate, σ(xV), which functions as a soft input filter. Empirically, this architecture yields superior loss performance at matched FLOP budgets when compared to traditional alternatives like GeLU or ReLU. Furthermore, SwiGLU facilitates smooth gradient flow, which helps prevent the issue of dead neurons during training. While the inclusion of a third weight matrix increases the parameter count, production stacks typically adjust the hidden dimension (dff) downward to compensate for the additional cost.