179 questions
No questions match those filters.
← All questions
EvaluationHard· LLM Judge· Retraining· Model Alignment
Your few-shot-calibrated LLM judge disagrees with your...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansRetraining a retriever against unverified labels risks optimizing the system to satisfy a flawed proxy. A 15% disagreement rate indicates a need for investigation rather than immediate action. You should perform a root-cause analysis on the disagreement cases to see if the judge’s prompt needs refinement or if the PM’s rubric is ambiguous. Retraining should only occur after the judge has been validated and the disagreement rate is brought within acceptable limits.