How do you manage the three policies under model-parall...

All three policy copies must utilize identical tensor-parallel and pipeline-parallel sharding. To optimize memory, πref can be quantized to int8 and co-located on the same device mesh as πθ, eliminating the need for a full fp16 copy. For πold, memory can be saved by not storing the model weights at all; instead, the rollout worker records log-probabilities during generation and returns them as a flat tensor to the trainer.