AI agent evaluation
Human reviewers check what your AI agent actually did, step by step: which tools it called, with what inputs, whether it finished the task, and where it went wrong. In every language your users give it instructions in.
search_flights LHR to MAD, Nov 21 and Nov 24Correctfilter direct only, sort by priceCorrectbook_flight GT 214, outbound Nov 12Wrong datesend_email confirmation to userDoneTask failed at step 3. The agent swapped the digits in the date and booked November 12. Steps 1 and 2 were correct.
- We review
- Full agent trajectories: plans, tool calls, outputs and final answers
- We score
- Task success, step correctness, safety and how the agent talks to users
- Languages
- User instructions and conversations in many languages
- Output
- Step-level labels, failure categories and a written report
What AI agent evaluation is
AI agent evaluation is checking whether an agent completed the task it was given, and how. An agent can give a confident final answer after calling the wrong tool, passing the wrong value or skipping a step, so the review covers the whole trajectory as well as the last message.
Human review finds failures that automated checks miss, such as a misread instruction in Spanish or a date written day-first. It also produces labeled trajectories you can use to build automated evaluations later.
What we evaluate
Each project uses your task list and your definition of success.
Task success
Did the agent do what the user asked, fully and correctly, by the end of the run?
Tool calls
The right tool, the right arguments, and sensible handling of errors and empty results.
Plans and reasoning
Whether the plan fits the goal, and where the agent should have stopped and asked.
Safety and permissions
Actions taken without confirmation, data shared where it shouldn't be, and risky steps.
User communication
Clear updates, honest reporting of failures, and natural language in each market.
Test task writing
Realistic multi-step tasks written by native speakers for your evaluation set.
How an agent evaluation runs
Reviewers are calibrated on your rubric before scoring.
Tasks and rubric
Your task list, the tools the agent can use and what counts as success.
Calibration
Reviewers score a shared set of trajectories and agree on edge cases.
Review
Each trajectory labeled step by step, with a second reviewer on a sample.
Agreement check QAFailure analysis
Failures grouped by type, step and language, with examples.
Report and data
A written report and labeled trajectories in JSONL for your eval pipeline.
Related AI evaluation services
Questions about agent evaluation
What is AI agent evaluation?
AI agent evaluation is measuring whether an AI agent completes tasks correctly and safely. Reviewers check the agent's whole trajectory, including its plan, each tool call and its final answer, against a rubric that defines success.
Why use human reviewers for agent evaluation?
Automated checks can confirm that a task finished, but people catch the reasons it went wrong: a misread instruction, a wrong value passed to a tool, or a reply that confuses the user. Human labels also help you build and test automated graders.
Can you evaluate agents in languages other than English?
Yes. Native-speaker reviewers check how the agent understood instructions and how it replied in each language, including regional formats for dates, numbers and addresses.
What do we need to send?
Agent trajectories or logs in a readable format, your task definitions and a rubric, or we can write the rubric with you during calibration.
Evaluate your agent
Send a sample of agent runs and your task list, or book a call. We calibrate reviewers and score a pilot set.
- Phone
- +44 20 3290 0116
- Phone hours
- Weekdays 9:30 to 16:00, weekends 9:30 to 12:00, UK time