RLHF and fine-tuning data
Prompts, model answers and preference rankings written by native speakers, for supervised fine-tuning and reinforcement learning from human feedback. In English and in the other languages your model has to handle.
PromptExplain compound interest to a 12-year-old in two sentences.
Compound interest is the addition of interest to the principal sum of a loan or deposit, calculated on accumulated interest over successive periods.
When you save money, the bank pays you a little extra called interest. Next year you earn interest on that extra too, so your savings grow faster over time.
It's when your money makes money. The longer you leave it, the more it grows.
Why B first: clear, accurate and two sentences. C is friendly but skips how interest grows. A is accurate but written for an adult.
Example ranking with a written reason.
- Data types
- Prompts, written answers, rankings, ratings and rewrites
- Writers and raters
- Native speakers, with subject experts where the prompts need them
- Output
- JSONL with prompt, chosen and rejected fields, or your schema
- Starts with
- A calibration round on your guidelines
What RLHF is
RLHF (reinforcement learning from human feedback) trains a language model on human judgments. People compare the model's answers and say which is better, and the model learns to prefer answers like the ones people chose. Supervised fine-tuning usually comes first: people write example answers for the model to imitate.
Both depend on who writes and who judges. A ranking from someone who isn't fluent in the language, or doesn't know the subject, teaches the model the wrong lesson.
Fine-tuning and preference data we produce
For instruction tuning, RLHF, DPO and reward model training.
Prompt writing
Realistic prompts in each language, spread across the tasks, topics and difficulty levels in your taxonomy.
Written answers
Model answers written from scratch by native speakers, following your style and length rules, for supervised fine-tuning.
Preference rankings
Pairs or sets of model answers compared and ranked, with ties allowed where neither answer is better.
Ratings and reasons
Scores on each dimension you define, with a short written reason, useful for training and checking reward models.
Rewrites
The model's own answer edited until it's right, giving you a matched pair of a flawed answer and a corrected one.
Multilingual and mixed-language data
Prompts and answers written natively in each language, including the mixed-language messages people really send, such as Hindi and English together.
What makes preference data useful
The difference between data that improves a model and data that adds noise.
- Clear guidelinesExamples for each judgment call, such as when a shorter answer should beat a longer one.
- Calibrated ratersEveryone scores the same items first, and we fix the guidelines where they disagree.
- Written reasonsA sentence on why one answer won, so you can audit the data and find guideline gaps.
- Honest tiesRaters can mark two answers as equal, rather than inventing a preference.
- Varied promptsTasks, topics and difficulty spread to match how people actually use your model.
How an RLHF data project runs
Agreement between raters is measured before production and on every batch after.
Guidelines
You share the task taxonomy, style rules and rating dimensions. We add examples for the hard calls.
Calibration
Writers and raters work the same small set, and we compare their output with yours.
Agreement review
Where people disagree, we refine the guidelines with you until they apply them the same way.
Your sign-off QAProduction
A share of items is double-rated, and a senior reviewer checks a sample of every batch.
Delivery
Data in your schema, with agreement figures and the guideline changes made along the way.
Formats
- Fine-tuning data
- JSONL with prompt and response fields, or chat format with system, user and assistant turns
- Preference data
- JSONL with prompt, chosen and rejected fields, plus ratings and reasons if you want them
- Platforms
- Your annotation platform, or files exchanged through your storage
- Languages
- English and major world languages, plus smaller languages confirmed per project
Formats, platforms and subject-expert pool to confirm
Related services
Questions about RLHF data
What is RLHF?
RLHF, or reinforcement learning from human feedback, is a way of training a language model using human judgments. People compare the model's answers to the same prompt and pick the better one, and the model is trained to prefer answers like those.
What's the difference between fine-tuning data and preference data?
Fine-tuning data is example answers written by people, which the model learns to imitate. Preference data is people's judgments about which of several model answers is better, which the model learns to prefer. Most projects use both.
Do you write prompts and model answers for fine-tuning?
Yes. Native speakers write prompts across your task taxonomy, and writers with the right background write the answers, following your style and length rules.
Can raters rank answers in languages other than English?
Yes. Each language is rated by native speakers, so judgments about fluency, tone and correctness come from people who would notice the difference.
Is DPO data the same as RLHF data?
Both use human preferences. DPO trains directly on pairs of chosen and rejected answers without a separate reward model. The data we deliver works for either: each prompt with a chosen answer, a rejected answer and, if you want, a rating and a reason.
How do you keep rankings consistent?
Every rater goes through calibration on your guidelines. During production, a share of items is rated twice and a senior reviewer checks a sample of each batch. You get the agreement figures with every delivery.
Start with a calibration round
Send your guidelines and a sample of prompts. We run a calibration round with native speakers and quote for production.
- Phone
- +44 20 3290 0116
- Phone hours
- Weekdays 9:30 to 16:00, weekends 9:30 to 12:00, UK time