DPO has no explicit reward model. Where does the reward...

In DPO, the reward model is implicit and is represented by the log-ratio beta * log(pi_theta(y|x) / pi_ref(y|x)). This ratio acts as the implied reward for any completion under the optimal policy. You can extract this scalar at inference time to rank completions, similar to an explicit reward model. However, because this reward was never trained with a classification head, it is not calibrated in absolute terms, though it remains well-ordered within the context of a specific prompt, making it useful for techniques like best-of-n reranking.