When would you use ZeRO-3 instead of pipeline paralleli...

ZeRO-3 shards parameters across DP ranks, which is highly effective if the DP group is large enough and the all-gather latency is acceptable. For a 70B model on 8 nodes, ZeRO-3 can shard parameters to roughly 2 GB per GPU, eliminating the need for pipeline stages and their associated bubble overhead. However, ZeRO-3 relies on frequent all-gather communication, so it is best suited for fat-tree fabrics, whereas pipeline parallelism is more robust when dealing with limited cross-chassis bandwidth.