How would you design a data curation pipeline that automatically identifies SFT examples likely to cause hallucination, at the scale of millions of examples?

The pipeline should utilize a self-consistency filter. For each candidate SFT example, sample k completions (e.g., k=5) from the base model using only the instruction. Compute agreement across these samples using a lightweight verifier or embedding similarity. Examples where base model completions show high agreement with the reference response are considered safe. Examples where the base model is inconsistent or contradicts the reference are flagged as hallucination risks. This approach is computationally efficient enough to run on GPUs at scale.