How do you call LLM or embedding APIs efficiently from...

Four layers, roughly in order of how much they help. Reusing a requests.Session() across calls instead of opening a new TCP connection every time gives a real but modest speedup. The bigger win for anything batch-shaped — embedding 10,000 documents, say — is switching to aiohttp with asyncio.gather(), wrapped in a semaphore that caps how many requests run concurrently so you don’t blow past rate limits. That’s the change that turns an hour-long sequential job into minutes.

Underneath both of those, retry logic with exponential backoff is not optional: LLM and embedding APIs fail intermittently for reasons unrelated to your request, and a pipeline with no retry logic crashes on the first transient 500 or timeout — three attempts with 1s, 2s, 4s delays is a reasonable default. Finally, for anything that doesn’t need a same-second response, batch API endpoints from providers like OpenAI and Anthropic trade latency for a real cost discount, which matters at scale for things like nightly re-embedding jobs.