← All questions
EvaluationHard· Faithfulness· Metrics· System Debugging

After a prompt change, faithfulness rose from 0.70 to 0...

When faithfulness increases, the system becomes more strictly grounded in the provided context. If the corpus is stale or contains errors, the model is forced to inherit those errors or abstain from answering entirely.

This creates a paradox where the system appears more ‘faithful’ to the provided documents, but the actual utility of the system drops because it is no longer able to provide correct information that exists outside of the retrieved context. The rise in complaints suggests that the ‘grounding pressure’ is causing the model to fail on queries where the corpus is insufficient, effectively trading off answer coverage for strict adherence to potentially flawed or outdated data.

Code playground (Python)