How would you evaluate a model once it’s in production?

Evaluating a model in production requires a dual approach. First, use aggregate telemetry to monitor ongoing health and performance. Second, implement shadow evaluation to test model upgrades against real-world traffic before full deployment. It is critical to stratify data by intent clusters and distinguish between proxy metrics, such as thumbs-up ratings or session continuation, and direct quality scores to get a complete picture of model performance.