← All questions
LLMOps & ProductionHard· Evaluation· Debugging· Model Upgrades

Accuracy dropped six points after a generator upgrade w...

Start by ruling out the retriever; since recall@k is computed against the retriever and it remains unchanged, it is likely not the source of the drop. Next, examine parse and refusal rates, as format-induced invalid outputs are frequently scored as wrong answers. Finally, perform a sweep of 20 formats on both the old and new models. The diagnostic goal is to determine if the incumbent format performed well on the old model but poorly on the new one, rather than simply comparing aggregate points.

Code playground (Python)