RAG evaluation services
Human reviewers judge both halves of your retrieval-augmented generation system: whether the retrieved passages are relevant, and whether every claim in the answer is supported by them. In the languages your users ask questions in.
- Retrieval
- Passage relevance and ranking for each question
- Answers
- Claim-by-claim grounding, completeness and citations
- Questions
- Realistic test questions written by native speakers
- Output
- Labeled test sets, scores per language and a failure report
What RAG evaluation is
RAG evaluation is measuring how well a retrieval-augmented generation system finds the right information and uses it. A RAG system can fail at retrieval, by fetching the wrong passages, or at generation, by adding claims the passages don't support. Good evaluation separates the two, so you know which part to fix.
Human labels give you a trusted reference for both. They're also the data you need to check whether an automated LLM judge agrees with people before you rely on it.
What we evaluate
Against your documents, your questions and your definition of a good answer.
Retrieval relevance
Each retrieved passage rated for how well it answers the question, so you can measure precision and ranking.
Answer grounding
Every claim in the answer checked against the passages: supported, unsupported or contradicted.
Completeness
Whether the answer covers what the user needed, or misses information the sources had.
Citations
Whether each citation points to a passage that actually supports the sentence.
Cross-language retrieval
Questions in one language answered from documents in another, checked by bilingual reviewers.
Test set writing
Realistic questions with reference answers and source passages, written from your documents.
How a RAG evaluation runs
Reviewers read your source documents as well as the answers.
Scope
Your document set, sample questions, languages and scoring rubric.
Test set
We write or collect questions and, if needed, reference answers.
Calibration
Reviewers score a shared sample and agree on what counts as supported.
Agreement check QAReview
Retrieval and answers labeled for each system version you send.
Report
Scores per language and question type, with examples of each failure.
Related AI evaluation services
Questions about RAG evaluation
What is RAG evaluation?
RAG evaluation measures how well a retrieval-augmented generation system works. It checks whether the system retrieves relevant passages for a question, and whether its answer is supported by those passages, complete and correctly cited.
Why use human reviewers instead of an LLM judge?
An LLM judge is fast, but you need to know it agrees with people first. Human labels give you that reference, and they catch subtle failures such as an answer that sounds right but adds a detail no source contains.
Can you evaluate RAG systems in other languages?
Yes. Native-speaker reviewers judge questions and answers in each language, including cross-language cases where the question and the documents are in different languages.
What do you need from us?
Access to the source documents or passages, a set of questions or logs, the system's retrieved passages and answers, and your rubric. We can write the rubric and test questions with you.
Evaluate your RAG system
Send sample questions, retrieved passages and answers, or book a call. We calibrate reviewers and score a pilot set.
- Phone
- +44 20 3290 0116
- Phone hours
- Weekdays 9:30 to 16:00, weekends 9:30 to 12:00, UK time