LLM evaluation in many languages
Native speakers score your model's answers against your rubric, compare outputs side by side, and tag the errors automated metrics miss: wrong facts, unnatural phrasing and answers that don't fit the market.
Prompt¿Cuánto tarda un envío a Monterrey?
- Accuracy
- 5
- 5
- Fluency
- 5
- 4
- Locale fit
- 5
- 2
Prefer A. B uses “vuestro”, a form readers in Mexico don't use.
Example side-by-side review with rubric scores from 1 to 5.
- Evaluation types
- Rubric scoring, side-by-side preference and error tagging
- Raters
- Native speakers, with subject experts where the content needs them
- Output
- Scores, labels and rater comments in CSV or JSON
- Starts with
- A calibration round on your rubric
What LLM evaluation covers
LLM evaluation is how you judge whether a large language model answers real prompts well. Human raters score each answer against a rubric, compare answers from different models or versions, and explain the problems they find.
Automated metrics and model-graded scoring cost less per item, but they miss things a native speaker catches at once: the wrong register, a word nobody in that market uses, a date in the wrong format, or an answer that's polite in English and blunt in Japanese. Our scores show you how far to trust those methods in each language.
What our raters check
Pick the dimensions that matter for your product. We write rater guidance for each one and calibrate before scoring starts.
Accuracy and invented facts
Raters check each claim against the source or a reference answer, and flag invented facts, wrong numbers and citations that don't support the claim.
Fluency and naturalness
Grammar, word choice and register, judged by native speakers of each language and regional variety you ship to.
Locale and cultural fit
Dates, currencies, units, names, forms of address, and topics that land differently in each market.
Instruction following
Whether the answer does what the prompt asked, including length, format, tone, language and any other limits you set.
Safety and tone
Harmful, biased or unsuitable answers flagged against your policy in each language. For adversarial testing, see red teaming.
Retrieval and citations
For retrieval-augmented systems, whether the retrieved passages support the answer and each citation points to the right source.
How the scoring works
Four formats, used on their own or together on the same test set.
- Rubric scoringEach answer is scored on each dimension, on a 1 to 5 scale or your own, with a comment on every low score.
- Side-by-side preferenceRaters see two answers to the same prompt, pick the better one and say why. Used for model comparisons and preference data.
- Error taggingProblems are tagged in the text by type, such as wrong fact, wrong locale or ignored instruction, so you can count them per language.
- Test set writingNative speakers write prompts and reference answers for each market, so your test set isn't translated English.
How an evaluation project runs
Most disagreement between raters comes from rubric wording, so we fix that before production.
Rubric
You share the rubric, the prompts or test set and the languages. We point out wording raters may read differently.
Calibration
Raters score the same small set. We compare their scores with each other's and with yours.
Agreement review
Where scores differ, we refine the rubric with you until raters apply it the same way.
Your sign-off QAProduction
Raters work through the set, with a share of items scored twice so agreement stays visible.
Report
Scores, labels and comments by language, with agreement figures and examples of each error type.
Formats and languages
- What you send
- Prompts and responses in CSV or JSONL, or access to your evaluation platform
- What you get
- Item-level scores, preference labels, error tags and comments in CSV or JSON
- Scales
- 1 to 5, pass or fail, pairwise preference, or your own scale
- Languages
- European, Middle Eastern, South Asian and East Asian languages, plus smaller languages confirmed per project
Formats and platforms to confirm
What you get back
Results you can feed into your training loop or show to the people deciding whether a model ships.
- Item-level scores. Every answer, every dimension, every rater.
- Rater comments. A short reason for each low score or preference.
- An agreement report. How often raters agreed, per language and per dimension.
- An error summary. Counts of each error type by language, with examples.
Related services
Questions about LLM evaluation
What is LLM evaluation?
It's the process of measuring how well a large language model answers real prompts. Human raters score answers against a rubric, compare answers from different models or versions, and tag specific errors, so you can see where the model is good enough to ship and where it isn't.
How do you score answers against a rubric?
We turn your rubric into rater guidance with examples for each score, run a calibration round, and adjust the wording until raters agree. Then each answer is scored on each dimension, with a comment on every low score.
How do you keep raters consistent?
Every rater goes through calibration on your rubric. During production, a share of items is scored twice, we track agreement by rater and language, and a senior reviewer settles disagreements and updates the guidance.
Can you test fluency and cultural fit?
Yes. That's where native speakers add the most. Raters judge register, word choice, formats and cultural fit for the specific market, such as Mexican rather than European Spanish.
How is human evaluation different from LLM-as-a-judge?
A model can score answers quickly and cheaply, but it shares many blind spots with the model being tested, especially in languages with less training data. Human scores on a sample tell you how far to trust the automated judge in each language.
Do you write test prompts too?
Yes. Native speakers write prompts and reference answers for each market, covering the tasks and topics your users bring, so you aren't testing on English prompts that were translated.
Test your model in the languages you launch in
Send your rubric and a sample of prompts and answers. We run a calibration round and quote for the full evaluation.
- Phone
- +44 20 3290 0116
- Phone hours
- Weekdays 9:30 to 16:00, weekends 9:30 to 12:00, UK time