How do you track and optimize LLM costs in production?

Cost tracking only becomes actionable once it’s granular enough to point at a specific decision. Logging total spend tells you the bill is high; logging spend by model, by feature, and by user tells you which feature is eating the budget and which model tier is driving it, which is what actually lets you prioritize where to optimize first rather than cutting costs uniformly and hoping it helps.

Once you know where the spend concentrates, the levers themselves are fairly standard: route more of that feature’s traffic to a cheaper model if quality allows it, add semantic caching if the queries repeat often, compress the prompt and any retrieved context to cut input tokens, cap output length where verbose responses aren’t needed, and take advantage of provider-side prompt caching for repeated system prompts. Combined, these can meaningfully cut a feature’s cost without touching the model it uses at all — but knowing which feature to target first is what the granular tracking actually buys you.