Why can’t you just pad all sequences to the same length...

Standard padding forces all sequences in a batch to match the length of the longest sequence. This is highly inefficient because the attention mechanism performs quadratic computation on padded positions that do not contribute to the output. Additionally, it leads to significant memory waste in the KV cache and creates ‘stragglers’ where short sequences are blocked by a single long sequence, often resulting in effective utilization rates as low as 12%.