You removed 8 of 32 layers from a 15B model and see a 1...

A uniform drop across MMLU categories suggests general capacity loss that can be addressed through distillation. However, if the drop is concentrated in specific areas like multi-hop reasoning or world knowledge, it indicates the removal of layers responsible for key associative recall. In such cases, cosine similarity is an insufficient metric for layer importance. Instead, you should use Taylor sensitivity scores to identify and swap back in the most critical layers before proceeding with distillation.