A lab claims their MoE model has 70B parameters and matches a 13B dense model in quality. What would you ask before accepting that comparison?

First, clarify if 70B is the total or activated parameter count. If it is 70B total with 7B activated, the training compute comparison is coherent, but the memory requirements differ significantly from a 7B dense model. Second, determine if the quality comparison is FLOP-controlled (same training compute) or size-controlled (same total parameters); beating a 13B dense model at equal training FLOPs is expected, whereas beating a 70B dense model at equal total parameters would be a much stronger result.