Does ZeRO reduce activation memory?

A common mistake is conflating model state with activation memory. ZeRO is specifically designed to shard model parameters, gradients, and optimizer states. Activation memory is determined by the batch size and sequence length and remains local to each worker, meaning ZeRO provides no direct reduction for these memory costs.