Your LLM feature costs too much at production volume. W...

First, stop calling the model at all where you don’t need to. Most volume in most LLM-backed systems doesn’t actually need the LLM — caching identical or near-identical requests, deterministic rules for the obviously clear-cut cases, and cheap classifiers for coarse triage handle the bulk of it before a large model ever needs to run.

After that, in order:

1. Route what remains to the cheapest model that still passes evaluation for that specific stage — not one model for everything.
2. Cut input tokens through better retrieval and tighter prompts, which usually improves quality at the same time it cuts cost.
3. Only then reach for quantization or distillation on whatever model is left carrying the load.

The ordering matters: the first lever is usually worth an order of magnitude more than the last one, and starting with quantization because it’s the “AI-sounding” answer skips the cheapest wins.