Custom LLM benchmarks and evaluation sets

Private test sets written from scratch by native speakers and subject experts, so you can measure your model on the languages, tasks and difficulty levels you care about. Every item has a checked answer key, and none of it has been published online for models to learn from.

LanguageItemSkillLevel
HindiWork out the tax in a price that already includes itMathHard
Arabic (Egypt)Explain what a common saying means in contextCultureMedium
SwahiliAnswer questions about a local news paragraphReadingMedium
PolishPick the reply a customer would find politePragmaticsHard
2 reviewersagree on every answer key
Originalwritten new, not copied from the web
Privatedelivered only to you

Example items from a multilingual benchmark. Real items include the full prompt and answer key.

Written by
Native speakers and subject experts, from scratch
Includes
Prompts, answer keys, rubrics, difficulty and skill tags
Checked by
A second reviewer on every item and answer key
Ownership
Private to you, never published

Why build a private benchmark

Public LLM benchmarks are useful for comparing models, but they have two problems. Their items are online, so newer models may have seen them during training, and they rarely test the languages, tasks and domains your product depends on. A private benchmark written for your use case tells you how your model will actually perform for your users.

We write the items, the answer keys and the scoring rubric, then run human scoring or hand the set to your automated pipeline.

What we build

Sized and balanced to the skills and languages you need to measure.

  • Multilingual test sets

    Items written natively in each language, not translated from English, so they test real language use.

  • Domain benchmarks

    Questions from law, medicine, finance, education or your own product, written by people who know the field.

  • Reasoning and math

    Multi-step problems with worked solutions, at the difficulty level you specify.

  • Open-ended tasks

    Writing, summarizing and advice tasks with rubrics and reference answers for human or model grading.

  • Safety test sets

    Prompts that probe harmful, biased or risky behavior in each language. More on red teaming.

  • Benchmark refresh

    New items each quarter, so the set stays useful as models improve.

How a benchmark is built

A pilot set comes first, so you can test the format on your models.

  1. Design

    Skills, languages, difficulty mix, item format and how answers will be scored.

  2. Pilot items

    A small set per language, run against your models to check difficulty.

  3. Writing

    Native-speaker writers and subject experts write items and answer keys.

  4. Review

    A second reviewer checks every item, answer and tag, and searches for any online copies.

    Second review QA
  5. Delivery

    The set in JSONL or your format, with a datasheet describing how it was made.

Questions about LLM benchmarks

What is an LLM benchmark?

An LLM benchmark is a fixed set of test items, such as questions or tasks with known answers or scoring rubrics, used to measure and compare how well language models perform on specific skills.

Why use a private benchmark instead of a public one?

Public benchmark items are published online, so models may have seen them in training, and they seldom cover your languages and use cases. A private set written for you gives a cleaner measure of how your model will perform.

Are the items translated from English?

No, unless you ask for that. Items are written natively in each language, so they test how people really write and the knowledge they share in that culture.

How many items do we need?

It depends on how many skills and languages you want to measure and how small a difference you need to detect. We recommend a size during the design step, and the pilot shows whether it's enough.

Scope your benchmark

Tell us the skills, languages and models you want to measure, or book a call. We propose a design and a pilot set.

Phone hours
Weekdays 9:30 to 16:00, weekends 9:30 to 12:00, UK time

Send us a message

We use your details only to reply to this message. See our Privacy Policy. To send files, use the quote request form.

Book a call

30 minutes with our project team. Choose a weekday and a time between 10:00 and 18:00 UK time.

Calls run Monday to Friday. Weekends aren't available.

Choose a date to see times