Custom LLM benchmarks and evaluation sets
Private test sets written from scratch by native speakers and subject experts, so you can measure your model on the languages, tasks and difficulty levels you care about. Every item has a checked answer key, and none of it has been published online for models to learn from.
| Language | Item | Skill | Level |
|---|---|---|---|
| Hindi | Work out the tax in a price that already includes it | Math | Hard |
| Arabic (Egypt) | Explain what a common saying means in context | Culture | Medium |
| Swahili | Answer questions about a local news paragraph | Reading | Medium |
| Polish | Pick the reply a customer would find polite | Pragmatics | Hard |
Example items from a multilingual benchmark. Real items include the full prompt and answer key.
- Written by
- Native speakers and subject experts, from scratch
- Includes
- Prompts, answer keys, rubrics, difficulty and skill tags
- Checked by
- A second reviewer on every item and answer key
- Ownership
- Private to you, never published
Why build a private benchmark
Public LLM benchmarks are useful for comparing models, but they have two problems. Their items are online, so newer models may have seen them during training, and they rarely test the languages, tasks and domains your product depends on. A private benchmark written for your use case tells you how your model will actually perform for your users.
We write the items, the answer keys and the scoring rubric, then run human scoring or hand the set to your automated pipeline.
What we build
Sized and balanced to the skills and languages you need to measure.
Multilingual test sets
Items written natively in each language, not translated from English, so they test real language use.
Domain benchmarks
Questions from law, medicine, finance, education or your own product, written by people who know the field.
Reasoning and math
Multi-step problems with worked solutions, at the difficulty level you specify.
Open-ended tasks
Writing, summarizing and advice tasks with rubrics and reference answers for human or model grading.
Safety test sets
Prompts that probe harmful, biased or risky behavior in each language. More on red teaming.
Benchmark refresh
New items each quarter, so the set stays useful as models improve.
How a benchmark is built
A pilot set comes first, so you can test the format on your models.
Design
Skills, languages, difficulty mix, item format and how answers will be scored.
Pilot items
A small set per language, run against your models to check difficulty.
Writing
Native-speaker writers and subject experts write items and answer keys.
Review
A second reviewer checks every item, answer and tag, and searches for any online copies.
Second review QADelivery
The set in JSONL or your format, with a datasheet describing how it was made.
Related AI evaluation services
Questions about LLM benchmarks
What is an LLM benchmark?
An LLM benchmark is a fixed set of test items, such as questions or tasks with known answers or scoring rubrics, used to measure and compare how well language models perform on specific skills.
Why use a private benchmark instead of a public one?
Public benchmark items are published online, so models may have seen them in training, and they seldom cover your languages and use cases. A private set written for you gives a cleaner measure of how your model will perform.
Are the items translated from English?
No, unless you ask for that. Items are written natively in each language, so they test how people really write and the knowledge they share in that culture.
How many items do we need?
It depends on how many skills and languages you want to measure and how small a difference you need to detect. We recommend a size during the design step, and the pilot shows whether it's enough.
Scope your benchmark
Tell us the skills, languages and models you want to measure, or book a call. We propose a design and a pilot set.
- Phone
- +44 20 3290 0116
- Phone hours
- Weekdays 9:30 to 16:00, weekends 9:30 to 12:00, UK time