# Evaluates an agent's quality systematically

> Competency building-agents/evaluates-agents

Page: https://schubergphilis.github.io/ai-training/competencies/building-agents/evaluates-agents/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

*Building agents* · competency `building-agents/evaluates-agents`

**Taught in:** the [Building agents](https://schubergphilis.github.io/ai-training/building-agents/) course

**Draws on:** [Evaluation and testing](https://schubergphilis.github.io/ai-training/topics/building-agents/evaluation/), [Guardrails and production](https://schubergphilis.github.io/ai-training/topics/building-agents/production/)

## Learning objectives

### Turns "good" into scorable criteria (base)

| Claim | Why | Example |
| ----- | --- | ------- |
| Each criterion in the rubric is something two people would score the same way on the same output. | A criterion that depends on the grader's taste measures the grader. | "Every claim in the answer is supported by a cited passage" replaces "the answer is well grounded". |
| The rubric is written from real outputs: good ones, bad ones and the arguments about them. | Criteria written from imagination miss the failures that occur and include failures that do not. | The learner reads thirty transcripts, notes what made the bad ones bad, and writes a criterion for each recurring kind. |
| Each criterion has a scale with anchored points, and the anchors are examples. | "Rate 1 to 5" without anchors gives different 3s from different graders and from the same grader on different days. | "Accuracy: 0 = contains a false statement, 1 = correct but incomplete, 2 = correct and complete", with one example transcript for each. |
| The learner checks the rubric by having a model grade with it and comparing the results with human grades on a sample. | A rubric that a model applies differently from a person is not ready to run at scale. | The model judge agrees with the human on 26 of 30 samples, so the learner rewrites the two criteria behind the four disagreements. |

Served by: [A model as the grader](https://schubergphilis.github.io/ai-training/building-agents/llm-as-judge/), [Turning good into a rubric](https://schubergphilis.github.io/ai-training/building-agents/rubrics/)

### Builds a representative input set with expected qualities (base)

| Claim | Why | Example |
| ----- | --- | ------- |
| The set covers the kinds of inputs that occur in use, in roughly their real proportion, plus the rare cases that matter most. | An agent tuned on easy or typical inputs looks good and fails on the input that costs the most. | The support agent set has 60 routine questions, 20 ambiguous ones, 10 in other languages and 10 that should be refused. |
| Each item records the expected qualities of a good answer instead of one exact answer. | Agents produce many correct answers, and matching one string fails all the others. | "Must mention the 30-day window, must not promise a refund, must offer the form link" instead of a reference reply. |
| The set grows from production failures and is versioned with the agent. | Every failure that reached a user is a test case the set was missing. | A user got a wrong answer about a holiday closure, so the question and the expected qualities go into the set with a link to the incident. |
| The learner keeps a part of the set that is never used for tuning. | A set the team iterates against becomes a set the agent has been fitted to. | Twenty percent of the items are held back and scored only at release time. |

Served by: [Reading failures one by one](https://schubergphilis.github.io/ai-training/building-agents/error-analysis/), [A golden set is the agent's regression suite](https://schubergphilis.github.io/ai-training/building-agents/golden-sets/)

### Grades the path the agent took, not only the final answer (base)

| Claim | Why | Example |
| ----- | --- | ------- |
| The learner scores the steps the agent took (which tools, in what order, with what arguments) as well as the final answer. | A correct answer reached by luck or by a dangerous path is a failure the final-answer score cannot see. | The agent answered correctly after calling the payment API three times with the wrong account, and the trajectory score fails it. |
| The learner writes down what a good path looks like for each task type: which tools it should use, which it should not, and roughly how many steps. | Without an expected path there is nothing to grade the actual path against. | "A lookup question should call search once and answer; more than three tool calls is a warning sign." |
| The learner reads a sample of full transcripts by hand every release, however good the scores. | Automated grading finds the failures it was built to find, and the new failure mode is in the transcripts. | The scores are unchanged, and the transcripts show the agent has started apologizing in every reply since the prompt change. |

Served by: [Reading failures one by one](https://schubergphilis.github.io/ai-training/building-agents/error-analysis/), [Recording and grading the path the agent took](https://schubergphilis.github.io/ai-training/building-agents/traces-and-trajectories/)

### Chooses what to fix from counted failures (base)

| Claim | Why | Example |
| ----- | --- | ------- |
| Before changing the agent, the learner reads the failed runs of a period, such as a week, and counts them by the kind of step where each went wrong. | Without a count, the team fixes the failure it remembers best, which is often the latest or the one a manager forwarded. | Of 20 failed runs, 11 are search misses, 4 are made-up amounts, 3 hit the step limit and 2 answer in the wrong language. |
| The learner fixes the most frequent kind first, with a change aimed at the part of the agent that failed. | A targeted fix can turn every failure of its kind into a pass, and a general change such as a larger model often does not reach a fault in a tool or in the search. | Five of ten failures are dates the tool could not read, so the learner writes the date format into the parameter description before touching the stop rule. |
| The learner checks the fix on the next batch of runs and counts the kinds again. | A fix can miss its target or break runs that passed, and once it works another kind becomes the most common. | In the week after the date fix, no run fails on a date the tool could not read, and search misses are now the largest kind, so they are the next fix. |

Served by: [Reading failures one by one](https://schubergphilis.github.io/ai-training/building-agents/error-analysis/)

### Chooses metrics that cannot improve without the real quality improving (expert)

| Claim | Why | Example |
| ----- | --- | ------- |
| For each metric, the learner can describe a way to raise it without making the agent better, and either closes that way or pairs the metric with one that catches it. | A metric that can be gamed will be, by the team or by the model, without anyone intending it. | "Answers per hour" is paired with "answers accepted by the user", because the first rises when the agent answers badly and fast. |
| The learner checks metric changes against a held-out set and against human judgment on a sample. | A rise on the set the team tunes against may come from the tuning rather than from the agent. | The rubric score rose five points, but on the held-out set it rose one, so the learner reports one. |
| The learner prefers metrics tied to the user's outcome over metrics about the agent's output. | Output metrics measure the agent's behavior, and the goal is the user's result. | "Ticket resolved without reopening within seven days" replaces "response contains the policy link". |

Served by: [A model as the grader](https://schubergphilis.github.io/ai-training/building-agents/llm-as-judge/)

### Grows the evaluation set from what goes wrong in real use (expert)

| Claim | Why | Example |
| ----- | --- | ------- |
| The learner logs the traces of real use from the first release and reads a sample of them on a schedule. | Failures that matter appear in real traces before anyone thinks to test for them, and only someone reading the traces finds them (Your AI Product Needs Evals, hamel.dev). | Every Friday the learner reads thirty traces from the week, and tags each failure with what went wrong. |
| Each failure found in use becomes a case in the evaluation set, with the expected quality written down. | A case taken from real use measures a problem users actually have, and keeps the fix from regressing. | A trace where the agent quoted an expired price becomes a test case that checks the price against the current list. |
| The learner removes personal and confidential data from a trace before it goes into the set. | An evaluation set is shared, versioned and kept for a long time, and a production trace can hold customer data. | The learner replaces the customer's name and address with made-up values and keeps the structure of the request. |

Served by: [Growing the evaluation set from traces](https://schubergphilis.github.io/ai-training/building-agents/evals-from-traces/)

## Alignment

| Framework | Code | Asks | Objectives here |
| --------- | ---- | ---- | --------------- |
| Ng, AI engineering skills map | Building and deploying AI applications | Build, evaluate, ship | writes-a-rubric, builds-a-golden-set, grades-trajectories, prioritizes-fixes, avoids-gamed-metrics |
| Brilliant Coding with AI | VER-7 | Measure quality and evaluate AI systems | writes-a-rubric, builds-a-golden-set, grades-trajectories, prioritizes-fixes, avoids-gamed-metrics |

## References

- Your AI Product Needs Evals, hamel.dev, https://hamel.dev/blog/posts/evals/