How do you build a per-response quality score for a RAG...

In the absence of labeled data, you can leverage the components already present in the RAG pipeline. By using an LLM as a judge, you can perform directed comparisons: checking the faithfulness of the answer against the retrieved context and the relevance of the answer to the user’s query. This provides a structured way to quantify quality without needing a gold-standard dataset.