# A golden set is the agent's regression suite

> Run a golden set of fifteen handbook questions with the qualities a good answer has, predict what a one-line prompt change breaks, hold one item in five back for release, and add two items.

Page: https://schubergphilis.github.io/ai-training/building-agents/golden-sets/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson we run a golden set for the handbook assistant after a
change to its prompt, and then add items to it. In
[Giving the agent a check it can run](https://schubergphilis.github.io/ai-training/coding-with-agents/self-checking-loops/)
you turned a manual check into a test that runs after every change.
**Golden sets** do the same for an agent. A golden set is a written list of
inputs, each with the qualities a good answer has, and a script runs all
of them every time the prompt, the model or a tool changes.

The agent is the basic retrieval agent from
[Retrieval as a tool the agent calls](https://schubergphilis.github.io/ai-training/building-agents/retrieval-as-a-tool/):
one keyword search over the twelve-document handbook, then a fake model
that answers from the passage it gets. The fake model reads its prompt,
so a change to the prompt changes what it does. Every run prints the
same result, and no application programming interface (API) key is
needed. The files are in
`site/examples/building-agents/golden-sets/`. Copy the `golden-sets`
folder, and each step below is `python3 golden.py <step>`.

## What a good answer has

A good golden set covers the questions the agent gets in normal use, the
unusual ones, and the ones it must not answer (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). A set with only
clear, well-formed questions doesn't tell you what happens with real users
(Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). This course adds a rule about the mix: take the proportions
from real use. A score on that mix says how the agent does on the
questions users ask.

Suppose the assistant's question log shows that a bit more than half of
the questions are routine, about a quarter are ambiguous, and the rest
ask about something the handbook doesn't cover. The fifteen items in
`golden-set.csv` follow that split, with eight routine items, four
ambiguous ones and three the agent should refuse. The file starts with
this header line:

```text
id,group,question,source,mentions,may_decline,held_out,origin
```

These are four of the items, with the first six columns lined up:

```text
id   group      question                                 source        mentions       may_decline
r01  routine    How many days of annual leave do I get?  leave.txt     25 days        no
a01  ambiguous  How much holiday can I take?             leave.txt     25 days        yes
a02  ambiguous  How many days can I take off?            leave.txt     25 days        yes
x01  refuse     Can I bring my dog to the office?
```

This course's rule is that each item records what a good answer has,
and it doesn't record one answer word for word. An agent can phrase a
correct answer many ways, and a check that compares with one reference
string fails all the others.
Here a good answer to r01 comes from `leave.txt` and contains "25 days".
For an ambiguous question, "not found" is also a good answer, because a
careful agent may decline a question it can't place. An answer from the
wrong page is never good, and for a question the handbook doesn't cover
"not found" is the only good answer. This is the check in `golden.py`:

```python
def passes(item: dict[str, str], result: dict) -> bool:
    """Checks the qualities the item expects, not one exact answer."""
    if result["answer"] == "not found":
        return item["group"] == "refuse" or item["may_decline"] == "yes"
    if item["group"] == "refuse":
        return False  # any answer to a question the handbook doesn't cover is a failure
    right_source = result["name"] == item["source"]
    return right_source and item["mentions"].lower() in result["answer"].lower()
```

Checks that look for a required string in the output, or for an answer
that must not be there, are cheap and give the same result every time
(Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). A question that needs judgment, such as whether an answer is
polite, needs a rubric and a grader, which the lesson
[Turning good into a rubric](https://schubergphilis.github.io/ai-training/building-agents/rubrics/) covers.

## Run the set

A short script that loads the set, runs each item through the agent and
checks the result is enough to start with (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). The `run` step does
that for every item that isn't held out, and prints the pass count per
group. Three of the fifteen items are held out, and the last section
explains why.

#### Example: The golden set on today's prompt

Run this, and compare what you see with the output below.

```sh
python3 golden.py run
```

Output:

```text
routine: 7 of 7
ambiguous: 3 of 3
refuse: 2 of 2
```

Every item passes. The count per group matters more than the total. A
total of 10 of 12 can mean two failed routine questions or two failed
refusals. Those two results need different fixes.

## Change one line, and predict what fails

The prompt around the retrieved passage is three lines, in
`PROMPT_LINES` in `agent.py`:

```python
PROMPT_LINES = [
    "The passages below were retrieved from the company handbook for this question.",
    "Answer from them. Quote the sentence you relied on.",
    "If the passages do not cover the question, answer exactly: not found.",
]
```

A colleague finds the last line too strict and removes it. An agent that
isn't told what to do when the handbook has no answer may try to answer
anyway (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). The fake model copies that behavior with a keyword rule.
With the last line in the prompt, it answers only when a sentence of the
passage shares two or more keywords with the question, and says "not
found" otherwise. Without the line, one shared keyword is enough. The line
has no effect on the search step, which gives every item the same passage
as before.

On today's prompt, four items pass because the agent says "not found".
For a01 the search returns no page. For a02, x01 and x02 it returns a
page, and no sentence on it shares two keywords with the question. Run
the golden set before the change goes in, and find out which items that
line was protecting. The `changed` step runs the set with the first two
lines only.

#### Checkpoint: Which items fail without the last line?

Predict the pass count per group, and the id of each item that fails,
when the golden set runs with the last line of `PROMPT_LINES` removed.
The script prints one `failed <id>` line per failure. Then run it.

```sh
python3 golden.py changed
```

The `answers` step prints the question and the answer for each item that
failed.

#### Example: What the agent answered

Run this, and compare what you see with the output below.

```sh
python3 golden.py answers
```

Output:

```text
failed a02: How many days can I take off?
  answer: Claims are filed within 30 days.
failed x01: Can I bring my dog to the office?
  answer: Core hours, when everyone in the office is reachable, are 10 to 15.
failed x02: What is my manager's salary?
  answer: Requests go to your manager at least two weeks ahead.
```

The routine items all pass, because each of them retrieves the right page
and the answer sentence shares two or more keywords with the question.
The items that fail are the ones where "not found" was the good answer
and a passage came back anyway. For a02, four pages contain "days", and
none contains "take" or "off". That's a tie, and the search takes the
first file in alphabetical order, `expenses.txt`, ahead of `leave.txt`.
The agent now answers with the sentence about claims. Item x01 quotes the
same sentence that run eight used to say yes to a dog in
[Turning good into a rubric](https://schubergphilis.github.io/ai-training/building-agents/rubrics/). Item a01 still
passes, because its search returns no passage at all. With no passage,
the agent has no sentence to answer with.

A person who tries the new prompt on a few routine questions sees
nothing wrong. The golden set shows that the line did its job on three of
the twelve items, all of them in the groups that a quick manual try
leaves out.

## Hold some back, and add past failures

A team that runs the same set after each change learns its items. It
rewrites the prompt until a02 passes, then until x02 passes, and after a
few months the prompt fits those twelve questions. While the score stays
where it was, it says less and less about the questions users ask. This
course keeps
about one item in five out of that loop: the held-out items run only
before a release, and nobody changes the agent while looking at them. In
`golden-set.csv` they have `held_out` set to `yes`, one per group.

The `release` step runs all fifteen items and counts the held-out ones
on their own line.

#### Example: The release run

Run this, and compare what you see with the output below.

```sh
python3 golden.py release
```

Output:

```text
routine: 7 of 8
ambiguous: 4 of 4
refuse: 3 of 3
held out: 2 of 3
failed r08: Who approves a replacement laptop for the payments team?
  answer: The cost center owner of your team approves a replacement laptop before that, see the team pages.
```

The development run said 12 of 12. The held-out item r08 needs two
lookups, the laptop page and then the payments team page, and one
search only finds the first. This is the failure the retrieval lesson
fixed with a loop. Twelve passes on the tuned part and a failure on the
held-out part is the signal that the development score was too
optimistic. When the team fixes r08, it has looked at that item, so it
moves r08 into the development part and holds back a new question from
the log in its place.

The other way a set grows is from failures that reached a user. When a
failure shows a case the set doesn't cover, the question goes into the
set (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). This course also records where the item came from. Item
x01 is one of those, and its `origin` column points at the run it came
from:

```text
x01,refuse,Can I bring my dog to the office?,,,no,no,run t08 of the rubrics lesson: the agent said yes to a dog
```

In a real team the origin is a link to the incident or the ticket, so a
later reader can see why the item exists before they delete it. A set
that never changes goes stale as the agent and its users change, so add
items as new kinds of question appear (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course).

#### Checkpoint: Which item goes in the held-out part?

A team has four candidate items for its golden set. Which one should go into
the held-out part?

- The question from last week's incident, which the team is fixing in the prompt this week.
- The ambiguous question the prompt was rewritten last month to pass.
- A routine question taken at random from this month's log, which nobody has run yet.
- A copy of a development item with the same question, so the two parts measure the same thing.

#### Pitfall: A set tuned until it passes

A team runs its golden set after every prompt change and fixes each item
that fails, until the set passes 40 of 40. The release goes out, and
users get wrong answers to questions like the ones the set held a month
ago. The prompt had been rewritten around those 40 questions, and the
set had stopped measuring anything else. A score on items the team
tuned against says how well the agent fits those items. Keep a part of
the set out of the tuning loop and run it only before a release. When
you use a held-out item to fix a failure, replace it with a new one.

## Exercise

Copy the `golden-sets` folder and add two rows to `golden-set.csv`. The
first comes from a past failure: pick a run in
`site/examples/building-agents/rubrics/transcripts/` where the agent
went wrong, write its question with the
source and the text a good answer mentions, and put the run in the
`origin` column. The second is a question the handbook doesn't cover and
the agent should refuse. Then set `held_out` so that three or four of
the seventeen items are held out, spread across the groups. Run
`python3 golden.py run` and `python3 golden.py release`. To see how the
agent handles your refused question without the not-found line, set its
`held_out` to `no` for a moment and run `python3 golden.py changed`,
then `python3 golden.py answers` to see what the agent said. Adding
items yourself shows what an item has to record for a script to check
it without you.

A good result prints one pass count per group, and the counts add up to
seventeen in the release run and to the items that aren't held out in
the development run. The release run has a `held out` line with three or
four items. Your past-failure item names its source page and a phrase
from the right answer, so it would fail if the agent repeated the old
mistake. Did your refused question pass because the model said "not
found" to the page it got, or because the search returned nothing?

Stretch: Add a third item that you expect to fail today, run the set, and write down in its origin column which lesson or change you expect to fix it.

## Recap

1. A golden set lists inputs with the qualities a good answer has, such
   as the source page and a phrase it must contain. A short script checks
   them after every change to the prompt or the model (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course).
2. Cover routine, unusual and out-of-scope questions (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course), in about
   the proportion users ask them, and read the pass count per group.
3. Run the set before a change goes in. A line in the prompt can protect
   items that a quick manual try never shows.
4. Hold about one item in five out of the tuning loop and run it only
   before a release. Replace a held-out item once you have used it to
   fix a failure.
5. Add every failure that reached a user as an item (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course), and
   record where it came from.

#### Habit

The next time an agent you work on gives a user a wrong answer, add the question to its test set before you fix the prompt.

## References

- Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/09-evaluating-and-testing-agents/
- Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/08-agentic-rag/