Explain speculative decoding. When does it help and whe...

Speculative decoding works by having a small draft model generate K token proposals in a single pass, which the large target model then verifies in parallel using a tree-structured attention mask. Accepted tokens are retained, while the first rejected token triggers a corrected sample from the target distribution to ensure exact distributional equivalence. The speedup is determined by the acceptance rate α, with latency reduction calculated as approximately K * α / (1 + K * c), where c is the relative cost of the draft model. This technique is most beneficial when tokens are highly predictable, such as in code or templated text, but it can hinder performance when the acceptance rate is low, as seen in creative or multilingual generation.