How do you choose the draft model for speculative decod...

The draft model must be significantly more efficient than the target model, ideally 10-20 times cheaper in parameter count or latency. It is crucial that the draft model is trained on the same data distribution and is tokenizer-compatible with the target model. In practical applications, self-speculative decoding—which utilizes early layers of the target model as the draft—is often used to bypass tokenizer compatibility problems and the memory overhead associated with maintaining a second, separate model. Advanced methods like EAGLE and Medusa further refine this by using learned auxiliary heads.