How do you choose dmodel and nlayers for a 13B paramete...

Following the 128-per-layer rule, a 13B model typically uses a dmodel of 5120 and 40 layers. Beyond these heuristic values, you must ensure that dmodel is divisible by the tensor-parallel degree to allow for efficient weight partitioning. Additionally, the number of layers must be divisible by the number of pipeline stages to ensure balanced workload distribution across the cluster.