What does proper logging look like for a production AI...

A normal service’s logs answer “did the request succeed and how long did it take.” An AI application’s logs need to answer a harder question: given that the request technically succeeded, was the actual output any good, and what did it cost to produce? That requires a different shape of log entry and a different discipline around what gets written where.

Every request needs a single ID that a tracing system like OpenTelemetry threads through the entire chain — the initial API call, the retrieval step, the LLM call, the final response — so a bad output can be traced back to whichever stage actually caused it, rather than staring at an opaque “response was wrong” report. Each stage logs its own structured fields: retrieval logs chunk count and top similarity score, the LLM call logs model name, prompt version, prompt and completion token counts, computed cost, and finish reason, since a response cut off by a length limit or blocked by a content filter needs completely different follow-up than one that completed normally.

PII scrubbing has to happen before logging, not as cleanup afterward — raw user messages routinely contain names, emails, or account details that shouldn’t sit in a log aggregator regardless of retention policy. Different error types also deserve different automated handling baked into the logging layer itself: a rate limit error retries, a content-filter trigger gets flagged for security review, and a context-length error gets truncated and retried rather than surfaced as a generic 500.