179 questions
No questions match those filters.
← All questions
EvaluationHard· Evaluation & Production
How do you use an LLM as a judge safely?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWith explicit awareness of its known biases, and never as the sole signal for anything that matters.
The three biases worth designing around:
- Position bias — the judge tends to favor whichever option it saw first. Randomize the order of options across evaluations.
- Verbosity bias — longer answers tend to score higher regardless of actual quality. Control for length, or explicitly instruct the judge to ignore it.
- Self-preference — a model tends to rate its own family’s output more favorably. Avoid judging a model’s output with a judge from the same model family that generated it.
Beyond mitigating those, calibrate the judge against real human labels on a representative sample before trusting its verdicts, and keep a human-labelled anchor set in the loop for anything commercially load-bearing — the judge is a scaling tool for evaluation, not a replacement for having ever checked against ground truth.