Where do you look first when a fine-tuned judge disagre...

Before assuming the model is deficient and jumping to retraining, investigate the data pipeline. Published error analyses show that a large percentage of discrepancies between human reviewers and automated judges are caused by the judge receiving different context or evidence snippets than the human. Logging the exact input provided to the judge is the first step in diagnosing these performance gaps.