Dataset quality audits
An independent check of training data you already have. Reviewers relabel a random sample, measure the error rate by label and language, and find the guideline gaps behind the mistakes.
- Delivery was fast and the box was fine.PositivePositive
- The app crashes every time I log in.NegativeNegative
- Oh great, another update that breaks my settings.PositiveNegative
- It's okay. Does what it says.NeutralNeutral
- Love waiting 40 minutes on hold. Really.PositiveNegative
- Refund arrived two days later.NeutralNeutral
- We audit
- Labeled text, audio, images, documents and preference data
- Also
- Synthetic data generated by models, before you train on it
- Method
- Random sample, blind relabeling, error analysis by cause
- Output
- Error rates, examples, guideline fixes and corrected labels
Why audit training data
A dataset quality audit measures how many labels in a dataset are wrong and why. Label errors teach a model the wrong thing and make test scores unreliable, and they are hard to see from inside the team that made them. An audit by reviewers who didn't build the data gives you a number you can trust and a list of what to fix.
Audits are useful before a large training run, when switching annotation vendors, after a guideline change, and before you rely on data generated by another model.
What an audit covers
Scoped to the decisions you need to make.
Label accuracy
Error rates overall, per label and per language, from a random sample relabeled blind.
Root causes
Each error traced to a cause: unclear guidelines, annotator mistakes, bad source data or tool problems.
Guideline review
Gaps and contradictions in your guidelines, with suggested wording and new examples.
Synthetic data checks
Model-generated data checked for factual errors, unnatural language, repetition and wrong labels.
Coverage and balance
Whether the data covers the languages, topics and edge cases your model will meet.
Correction
If you want, we fix the errors across the full dataset after the audit. More on managed annotation.
How an audit runs
Reviewers don't see the original labels while they work.
Scope
The dataset, guidelines, languages and questions you need answered.
Sample
A random sample, sized so the error rate is meaningful per label.
Relabel
Trained reviewers label the sample blind, with overlap.
Adjudicate
A senior reviewer settles each disagreement and records the cause.
Senior review QAReport
Error rates, examples, causes and recommended fixes.
Related AI data services
Questions about data quality audits
What is a dataset quality audit?
A dataset quality audit is an independent review of labeled data. Reviewers relabel a random sample without seeing the original labels, then the two sets are compared to measure the error rate and find the causes of errors.
How large a sample do you check?
It depends on how many labels you have and how precise the error rate needs to be. We size the sample with you so each label you care about gets enough items to measure.
Can you validate synthetic data?
Yes. Native speakers check model-generated data for factual errors, unnatural phrasing, repetition and wrong labels, and report how much is safe to train on.
Will you fix the errors you find?
If you want. We can correct the full dataset, or update the guidelines and retrain your existing annotators.
Audit a sample
Send the dataset or a sample with your guidelines, or book a call. We agree the sample size and the questions the audit must answer.
- Phone
- +44 20 3290 0116
- Phone hours
- Weekdays 9:30 to 16:00, weekends 9:30 to 12:00, UK time