What is the capacity factor in a mixture-of-experts mod...

The capacity factor is defined as capacity = bC * T / N, where C sets the limit on how many tokens an expert processes. Setting C higher improves representation quality by reducing token drops, but it requires larger pre-allocated buffers, which increases memory usage and latency. Conversely, setting C too low leads to frequent drops during imbalanced batches, while setting it too high wastes memory that could otherwise be used for activations or KV cache entries.