179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Latency· FiD· Amdahl's Law
We shipped FiD-Light and end-to-end latency fell 4%. Wa...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansDo not defend the number; decompose it. FiD-Light compresses only the decoder’s cross-attention cache, so its benefit is limited to 0.29L ms. If the encoder represents the vast majority of the budget, the total system gain will be minimal. Future improvements should focus on sharding the encoder across devices or reusing encoder outputs across queries for hot passages.