# Growing the evaluation set from traces

> Read a weekly sample of the handbook assistant's traces, label each run good or bad, find the failures the golden set has no case for, strip personal data from them, and add them as cases that the deploy gate runs on the next change.

Page: https://schubergphilis.github.io/ai-training/building-agents/evals-from-traces/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson we read a week of traces from an agent that staff use
every day and label each run good or bad. The failures of a kind that
the golden set has no case for become new cases. The previous lesson put a gate
in front of every change, and the gate ran 21 questions. Staff ask
hundreds a week, and some of their questions are unlike anything in the
set. **Trace-driven evaluation** brings those questions into the set. The agent logs every
run, and people read a sample of the traces. Each bad run of a new kind
becomes a case that the next change must pass.

The agent is the handbook assistant of
[Gating a deploy on the evaluation](https://schubergphilis.github.io/ai-training/building-agents/eval-gated-deploys/),
and the rollout of that lesson is finished. The shorter fixed prompt is
live for the whole office, and it is `prompts/current.txt` in this
lesson's copy. The gate, the golden set and the stricter candidate prompt are the
files of that lesson. The files are in
`site/examples/building-agents/evals-from-traces/`. Copy the
`evals-from-traces` folder, and run each step below from inside the copy,
as `python3 traces.py <step>` or `python3 gate.py <prompt>`. No
application programming interface (API) key is needed.

The traces in this lesson are invented, and so are the people and their
employee numbers. Real traces come from the trace store that a production
agent writes to, and they hold what real people typed, which can be
personal data.

## Log every run, and read a sample

Before people or a model can judge what an agent did, its runs have to be
logged (Your AI Product Needs Evals, hamel.dev). The assistant writes one trace per run. Its records
use the fields of
[Recording and grading the path the agent took](https://schubergphilis.github.io/ai-training/building-agents/traces-and-trajectories/).
A tool record holds the search, and a model record holds the answer. The
records here leave out the token counts, the latency and the cost, which
this lesson doesn't use, and they add `user`, the employee number of the
person who asked.

How much to read changes over time. Husain reads as much as he can when
a product is new, and reads a smaller sample as it matures, but he never
stops reading (Your AI Product Needs Evals, hamel.dev). This team picks 20 runs at random from each
week and reads them on Friday. He also says that reading must take as
little effort as possible, and he often builds his own viewer that puts
everything a reader needs on one screen (Your AI Product Needs Evals, hamel.dev). The `week`
step is this team's viewer. It prints each run of the sample with its
question and the answer the person got.

#### Example: The traces of week 1

Run this, and compare what you see with the output below.

```sh
python3 traces.py week 1
```

Output:

```text
week 1 sample: 20 runs
w1-01 Mon  How many days can I work from home?
      answer: Staff may work from home up to three days per week.
w1-02 Mon  What is the budget for training?
      answer: Every employee has a training budget of 1500 euros per year.
w1-03 Mon  Can I expense a taxi to the airport?
      answer: not found
w1-04 Mon  How many days of sick leave do I get?
      answer: Every employee accrues 25 days of annual leave per year.
w1-05 Tue  Who is the cost center owner of the payments team?
      answer: The cost center owner of the payments team is Maria Duarte, head of platform.
w1-06 Tue  What do I do with a phishing email?
      answer: Forward a phishing email to the security team and delete it afterwards.
w1-07 Tue  Where do I report a lost laptop?
      answer: Report a lost laptop to the service desk the same day.
w1-08 Tue  How many vacation days do I have?
      answer: not found
w1-09 Wed  How much do I get for my home office?
      answer: A home office allowance of 30 euros per month is paid with salary.
w1-10 Wed  Is there free coffee?
      answer: not found
w1-11 Wed  My wife is due in May, how many days of parental leave do I get?
      answer: Every employee accrues 25 days of annual leave per year.
w1-12 Wed  How far ahead are flights booked?
      answer: Flights are booked through the travel desk at least 14 days ahead.
w1-13 Thu  Who approves a new laptop for the data team?
      answer: The cost center owner of your team approves a replacement laptop before that, see the team pages.
w1-14 Thu  Where are passwords kept?
      answer: Passwords are kept in the company password manager.
w1-15 Thu  Can I carry over unused leave days?
      answer: Unused days carry over until 31 March.
w1-16 Thu  How fast are service desk tickets answered?
      answer: The service desk answers tickets within one working day.
w1-17 Fri  How much are meals reimbursed on a trip?
      answer: Meals on a work trip are reimbursed up to 40 euros per day with a receipt.
w1-18 Fri  When do I get my accounts as a new starter?
      answer: New starters get their accounts on day one from the service desk.
w1-19 Fri  Is there a gym at the office?
      answer: not found
w1-20 Fri  What are core hours?
      answer: Core hours, when everyone in the office is reachable, are 10 to 15.
```

#### Checkpoint: Why read traces when the gate passes?

Every Friday the team reads 30 traces from the week. A colleague asks
why, since the gate passes and the live rates are level. What's the
reason?

- To check the pass counts the gate printed, because a bug in the gate script could hide failures.
- To replace the gate over time, because real conversations say more than any fixed set can.
- To collect the replies customers liked, so the set can use them as the expected answers.
- To find failures that no item of the set checks, because the set holds only what someone thought to ask.

## Label each run good or bad

A trace has no expected answer. Nobody wrote one, because nobody knew the
question would come. The reader decides whether the run was good, here by
checking the answer against the handbook pages in `docs/`. Husain starts
with a label of good or bad for each trace, because finer scores are
harder to manage than two labels (Your AI Product Needs Evals, hamel.dev).

Sixteen of the twenty runs are good. Three of them answer "not found",
and that's right. The handbook says nothing about taxis or coffee, and
nothing about a gym. Runs w1-04, w1-08, w1-11 and w1-13 are bad. For each one, write down the first step that
went wrong, the way
[Reading failures one by one](https://schubergphilis.github.io/ai-training/building-agents/error-analysis/)
did:

| Run   | What went wrong                                                        |
| ----- | ---------------------------------------------------------------------- |
| w1-04 | no rule for sick leave, and the agent quoted the annual leave rule     |
| w1-08 | the handbook calls it "annual leave", and the search got expenses      |
| w1-11 | no rule for parental leave, and the agent quoted the annual leave rule |
| w1-13 | the answer needs a second lookup, the data team page                   |

Runs w1-04 and w1-11 are the worst of the four. A person who asked about
sick leave was told a rule that the handbook doesn't have. The question
shares two words with the annual leave sentence, "days" and "leave", and
two shared words are enough for the agent to answer.

## Which failures does the set already have?

In [Reading failures one by one](https://schubergphilis.github.io/ai-training/building-agents/error-analysis/) each
of the ten failures added an item. A weekly sample brings the same kinds
back week after week. So here a bad run of a kind that the set already
has adds to the count of that kind, and a bad run of a new kind becomes
a case. For each bad run, look in `golden-set.csv` for an item that
would fail on the same mistake, at the same step.

- w1-08 has the kind of a01, "How much holiday can I take?". In both runs
  the search doesn't return the page with the answer, because the handbook
  doesn't use the user's word. For w1-08 it returns the expenses page, and
  for a01 it doesn't find a page. The set accepts "not found" from a01, and the
  `helpful` metric already counts it as a failure.
- w1-13 has the kind of r08. The search returns the laptop page, and the
  answer needs a second lookup, the team page. r08 fails on every prompt
  today, and the routine threshold of the previous lesson allows it.
- w1-04 and w1-11 have no item of their kind. The refuse items x01
  to x06 all get "not found" today, because none of them shares two words
  with a sentence in the handbook. So the gate passed while staff got a
  made-up leave rule.

A failure of a kind that the set already has tells you how often that
kind happens, and the count helps you choose what to fix first. A new
kind tells you what the set is missing. This week has one new kind,
with two runs.

#### Checkpoint: New kind or known kind?

Buckets:

- The set has a case of this kind
- A new kind: add a case

Items:

- "How much time off can I take?": the search returned no page, because the handbook calls it annual leave
- "How many days of study leave do I get?": the handbook has no such rule, and the agent quoted the annual leave rule
- "Who approves a replacement laptop for the data team?": the search returned the laptop page, and the agent gave the general rule without the owner's name from the team page
- "Who do I tell about a lost lpatop?": the search returned the laptop page, which answers it, and the agent still said not found
- "When are claims for meals on a trip due?": the search returned the expenses page, and the agent quoted the meals sentence instead of the claims sentence

## Remove personal data before a trace becomes a case

The golden set is a file in version control. Everyone who works on the
agent reads it, and it is kept for years. A trace holds what a person
typed, and run w1-11 holds more than its question needs. The `show` step
prints its records:

#### Example: The trace of run w1-11

Run this, and compare what you see with the output below.

```sh
python3 traces.py show w1-11
```

Output:

```text
{"run": "w1-11", "span": 1, "kind": "tool", "name": "execute_tool search_docs", "tool": "search_docs", "args": {"query": "My wife is due in May, how many days of parental leave do I get?"}, "result": {"name": "leave.txt"}, "user": "emp-2613"}
{"run": "w1-11", "span": 2, "kind": "model", "name": "chat fake-model", "asked": "answer", "answer": "Every employee accrues 25 days of annual leave per year.", "user": "emp-2613"}
```

The employee number names a person, and the first half of the question
says something about that person's family that a colleague shouldn't
learn from a test file. Neither is needed to test the failure. Remove
the details that point at a person, and keep the structure of the
request: "How many days of parental leave do I get?". The case leaves out
the employee number, and its origin names the trace id. The trace id
leads back to the full trace only for people who may read the trace
store.

A rewritten question can also lose the failure. Check that the stripped
question still fails the same way on the live prompt before you add it:

#### Example: The stripped question on the live prompt

Run this, and compare what you see with the output below.

```sh
python3 traces.py ask current "How many days of parental leave do I get?"
```

Output:

```text
search returned: leave.txt
answer: Every employee accrues 25 days of annual leave per year.
```

The same page and the same wrong sentence, so the stripped case tests
what the trace showed.

#### Checkpoint: A trace with a health detail

A trace from the employee emp-5530 asks: "I need surgery in June, how
many days of sick leave do I get?". The assistant answered with the
annual leave rule. What goes into the golden set?

- Add the question as the person wrote it, because the set is internal and the detail shows why the person asked.
- Leave the trace out of the set, because a question that came with health data may never become a case.
- Use "How many days of sick leave do I get?", check that it still gets the annual leave rule, and add it with the trace id.
- Ask a model for a new question on the same topic, such as "What is the policy for illness absence?", and add that.

## Add the cases, and gate the next change

The two new cases go into `golden-set.csv` in the format of the golden
set lesson. They are refuse items, because the handbook covers neither
question. Their `held_out` is `no`, because the team will change the
agent while it looks at them, and
[A golden set is the agent's regression suite](https://schubergphilis.github.io/ai-training/building-agents/golden-sets/)
keeps such items in the part that runs after every change. The `origin`
column names the trace:

```text
x07,refuse,How many days of sick leave do I get?,,,no,no,trace w1-04: gave the annual leave rule as sick leave
x08,refuse,How many days of parental leave do I get?,,,no,no,trace w1-11 with personal details removed: gave the annual leave rule as parental leave
```

Add the two rows to the end of `golden-set.csv` in your copy, and run the
gate on the prompt that staff get today:

#### Example: The gate on the live prompt, with the new cases

Run this, and compare what you see with the output below.

```sh
python3 gate.py current
```

Output:

```text
prompt: current
  refused  hard  6 of 8    needs 100%  FAIL
  routine  hard  9 of 10   needs 90%   pass
  helpful  soft  11 of 15  needs 70%   pass
deploy: blocked
```

The prompt that passed the gate last week fails it now. The agent didn't
change, and the set did. The two cases fail on the live prompt, which
shows that they catch the failure from the traces. From now on the hard
gate blocks every deploy until a change passes them, even a change that
fixes something else. A team that doesn't want that adds the cases in the
same change as the fix. This team adds them now and accepts that nothing
else ships until the fix does. Staff get a made-up leave rule today. The
refusal gate needs 100% because any answer to a refuse item can be a
made-up policy, and that makes the fix the first change to ship. The next
change is the fix.

The previous lesson had a candidate that might be that fix. The stricter
prompt answers only when one sentence covers every keyword of the
question, and it waited for a sign-off because it lost the helpful answers
to a03 and a04. Run the gate on it with the new cases:

#### Example: The gate on the stricter prompt, with the new cases

Run this, and compare what you see with the output below.

```sh
python3 gate.py stricter
```

Output:

```text
prompt: stricter
  refused  hard  8 of 8    needs 100%  pass
  routine  hard  9 of 10   needs 90%   pass
  helpful  soft  9 of 15   needs 70%   FAIL
new failures: none
less helpful: a03 a04
deploy: waits for a person to sign off
```

The decision is still a person's, but the person now has more to weigh.
On one side are two helpful answers that the stricter prompt loses. On
the other side are two refusals that it gets right, from questions that
staff asked this week. Before the traces, the set had no case on the
other side. Next Friday the team reads a new sample, and from now on the
gate checks every change against a set that has these two cases.

#### Pitfall: Pasting the trace into the set as it is

A team adds a case from a trace in a hurry, with the question as the
person typed it: a name, a diagnosis and an employee number. Many people
read the golden set. A contractor reads it, and the file goes into a
copy for a vendor. Six months later someone notices and deletes the line,
and every earlier commit still holds it. Take out the details that point
at a person before the case is written. Then check that the stripped
question still fails.

The exercise uses the sample of the second week. The `week 2` step lists
it.

#### Example: The traces of week 2

Run this, and compare what you see with the output below.

```sh
python3 traces.py week 2
```

Output:

```text
week 2 sample: 20 runs
w2-01 Mon  How long is the onboarding checklist?
      answer: The onboarding checklist takes two weeks and ends with a review with your manager.
w2-02 Mon  Who do I tell about a lost labtop?
      answer: not found
w2-03 Mon  What is the default class for short flights?
      answer: Economy class is the default for flights under six hours.
w2-04 Mon  How often are laptops replaced?
      answer: Laptops are replaced every four years.
w2-05 Tue  When are claims for a work trip due?
      answer: Meals on a work trip are reimbursed up to 40 euros per day with a receipt.
w2-06 Tue  Who runs the warehouse?
      answer: The data team runs the warehouse and the reports.
w2-07 Tue  Can I carry over my training budget?
      answer: Unused days carry over until 31 March.
w2-08 Tue  Is home office equipment company property?
      answer: Equipment stays company property.
w2-09 Wed  Is there a parking garage?
      answer: not found
w2-10 Wed  What do I do with a fishing email?
      answer: not found
w2-11 Wed  Who builds the checkout?
      answer: The payments team builds the checkout and billing services.
w2-12 Wed  When is the building open on weekdays?
      answer: The building is open from 7 to 20 on weekdays.
w2-13 Thu  Can my dog come to the office?
      answer: not found
w2-14 Thu  When do unused leave days expire?
      answer: Every employee accrues 25 days of annual leave per year.
w2-15 Thu  Do I need a receipt for meals?
      answer: Meals on a work trip are reimbursed up to 40 euros per day with a receipt.
w2-16 Thu  How many euros is the training budget?
      answer: Every employee has a training budget of 1500 euros per year.
w2-17 Fri  Who is the cost center owner of the data team?
      answer: The cost center owner of the data team is Jonas Berg, head of analytics.
w2-18 Fri  Who books train tickets?
      answer: Train tickets are booked through the travel desk.
w2-19 Fri  Do conference tickets count against my budget?
      answer: Conference tickets count against the training budget.
w2-20 Fri  What is the home office allowance?
      answer: A home office allowance of 30 euros per month is paid with salary.
```

## Exercise

In your copy, with x07 and x08 added, read the sample of week 2 with
`python3 traces.py week 2`, and label each run good or bad against the
pages in `docs/`. For each bad run, write the first step that went wrong,
and look in `golden-set.csv` for an item of the same kind. For each new
kind, write one case per bad run in the person's words, without the
employee number, and with the trace id in `origin`. Check with
`python3 traces.py ask current "<question>"` that each case fails today.
Add the rows, and run `python3 gate.py current` and
`python3 gate.py stricter`. Twenty minutes is enough. A sample read this
way shows what the gate didn't know before.

A good result has five bad runs and one new kind. In w2-02 and w2-10 a
misspelled word, "labtop" and "fishing", leaves one word of the question
that the handbook has, "lost" and "email". The search returns the right
page on that one word, the laptop page and the security page. One shared
word is too few for an answer. The agent says "not found" to a question
that the page answers. That is the kind of a05, "Where do I keep my
passwords?", which gets the security page and "not found" on the one
word "passwords". Run w2-07 has the kind of x07 and x08, an uncovered
question answered with a rule that shares two words with it. These
three runs add to the counts of kinds that the set has. In w2-05 and
w2-14 the search also returns the right page, and the agent quotes the
wrong sentence on it. The set has no item of that kind. The two new
cases are routine items, with the page and a phrase from the right
sentence, such as `30 days` for w2-05. Week 2 has no personal data in its
questions, and the employee numbers stay out of the set. These are the
two rows of a good result:

```text
r13,routine,When are claims for a work trip due?,expenses.txt,30 days,no,no,trace w2-05: quoted the meals sentence
r14,routine,When do unused leave days expire?,leave.txt,31 March,no,no,trace w2-14: quoted the annual leave sentence
```

With x07, x08 and these two rows in the set, the two gate runs print
this:

#### Example: The gate on both prompts, with the cases of week 2

Compare this with the output of your own two gate runs.

```sh
python3 gate.py current
python3 gate.py stricter
```

Output:

```text
prompt: current
  refused  hard  6 of 8    needs 100%  FAIL
  routine  hard  9 of 12   needs 90%   FAIL
  helpful  soft  11 of 17  needs 70%   FAIL
deploy: blocked
prompt: stricter
  refused  hard  8 of 8    needs 100%  pass
  routine  hard  9 of 12   needs 90%   FAIL
  helpful  soft  9 of 17   needs 70%   FAIL
new failures: none
less helpful: a03 a04
deploy: blocked
```

The two new cases fail on both prompts. The routine metric drops to
9 of 12, and the gate blocks both. The kind of a05 got two more runs
this week, and the new kind has two runs. Which of the two would you fix
first, and why?

Stretch: Run python3 traces.py ask stricter with the question of each new case. Which of the new kinds of both weeks does the stricter prompt fix, and which part of the agent would a fix for the other change?

## Recap

1. Log the agent's traces, because people and models can only judge the
   runs that were logged (Your AI Product Needs Evals, hamel.dev).
2. Read a sample of the traces on a schedule, more at first and fewer
   later, and label each run good or bad (Your AI Product Needs Evals, hamel.dev).
3. For each bad run, look for an item of the same kind in the set. Each
   bad run of a new kind becomes a case, with the quality a good answer
   has written down and the trace id as its origin.
4. Take out the details that point at a person before a trace becomes a
   case, and check that the stripped case still fails on the live
   version.
5. A new case fails on the live version until a change fixes it, and the
   gate then checks that change against it.

#### Habit

The next time you check the dashboard of an agent you work on, open five of this week's traces and read them in full before you look at the scores.

## References

- Your AI Product Needs Evals, hamel.dev, https://hamel.dev/blog/posts/evals/