When should you use activation checkpointing vs. activa...

Activation checkpointing is applied at layer boundaries (such as matmuls or full transformer blocks) to reduce peak memory usage when the model does not fit in GPU RAM, though it carries a real runtime cost because these layers are compute-bound. In contrast, activation recomputation is applied inside memory-bound kernels (like elementwise activations or normalization) to speed up the backward pass. Because the compute units are idle during these memory-bound operations, the recomputation cost is effectively zero.