What are the main failure modes of pairwise preference...

Pairwise preference collection faces several failure modes. These include a strong length bias, where models or humans prefer longer responses regardless of quality; the inability to verify facts under speed pressure; AI judge self-preference bias; and annotator demographic skew, which can inadvertently embed cultural assumptions into the reward model.