179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Model Parallelism· Quantization· Memory Optimization
A 70B model is too large for your H100 fleet. How do yo...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFitting large models requires a combination of architectural and numerical strategies. Model parallelism distributes the model across multiple GPUs, while quantization reduces the memory footprint per parameter. By moving from BF16 to INT8, the memory requirement is halved, which is often the difference between needing a multi-node cluster and fitting the model on a single node. Using memory-safe quantization schemes ensures that the reduction in precision does not catastrophically impact model performance.