You are designing a 100B-parameter model for edge hardw...

SwiGLU requires two back-to-back GEMMs before the element-wise product, which can cause two memory round-trips to HBM, creating a throughput bottleneck on memory-constrained edge hardware. To mitigate this, one could use GeLU, which requires only one GEMM; rewrite the kernel to fuse both GEMMs into a single wider matmul; or use squared ReLU, which requires one GEMM and can exploit sparsity if the hardware supports it.