179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Model Compression· Quantization· Pruning
How would you compress a 70B model to fit on a single 8...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansA 70B model in BF16 requires approximately 140 GB, which exceeds the capacity of a single 80 GB H100. By applying structured pruning, you can reduce the model to roughly 35B active parameters, which brings the weight footprint down to about 70 GB. Further applying INT4 quantization reduces the footprint to approximately 35 GB, providing enough headroom for the KV cache. Finally, knowledge distillation from the original 70B teacher is used to recover accuracy lost during the compression process.