You fine-tuned with RAFT and your counterfactual errors...

The failure occurs because RAFT supervises one answer per example, spreading the credit assignment across all documents in the context. This is sufficient for a global ‘trust-the-context’ gate, but it does not allow the model to perform per-document evaluation. To fix this, you must introduce per-document defect labels before the answer. Since you cannot label counterfactuality from natural pairs, you must manufacture it by substituting entities to create documents with known falsity.