179 questions
No questions match those filters.
Throughput or latency — which are you optimizing?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThey trade against each other, so the honest answer is “depends which the use case actually needs” — and knowing which one matters is the actual skill, not defaulting to one.
Larger batches increase aggregate tokens-per-second across all concurrent users, but they increase time-to-first-token for any individual request, because that request now waits behind others sharing the batch.
- An interactive assistant — someone waiting on a reply — is a latency system. Optimize time-to-first-token and per-token latency, even at some cost to aggregate throughput.
- An overnight batch job processing a large volume with no one watching in real time is a throughput system. Maximize tokens per second across the whole run; individual-request latency barely matters.
Getting this backwards — batching aggressively on a chat product, or optimizing single-request latency on a bulk job — is one of the most common serving mistakes I see.