# A model as the grader

> Have a stand-in judge with a length bias grade runs of a handbook assistant, compare its scores with a person's, and pair numbers that rise on their own with ones that catch them.

Page: https://schubergphilis.github.io/ai-training/building-agents/llm-as-judge/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson a judge grades runs of the handbook assistant against a
rubric, and we check its grades against a person's before we trust
them. Then we look at numbers that rise while the agent stays the same,
and at the number that shows what a prompt change did.

The runs come from the handbook assistant of
[Turning good into a rubric](https://schubergphilis.github.io/ai-training/building-agents/rubrics/), the retrieval
agent that answers staff questions from a twelve-page handbook and must
quote the sentence it relied on. The sixteen runs are in
`site/examples/building-agents/llm-as-judge/runs/`: twelve in the
development part, which the team looks at while it changes the agent,
and four held out, as in
[A golden set is the agent's regression suite](https://schubergphilis.github.io/ai-training/building-agents/golden-sets/).
The author wrote the runs and the person's grades to show where a judge
and a person differ, so they're illustrative and don't come from a
model. Copy the `llm-as-judge` folder, and run each step below from
inside it as `python3 grade.py <step>`.

## A judge made of rules

A person can't read a thousand runs a day. **LLM as judge**, where LLM
stands for large language model, means a model grades the runs instead.
The judge gets the question, the agent's answer and the criteria, and
often a reference answer that lists what a good answer contains
(Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). It returns a score for each criterion, and it can grade
thousands of runs overnight.

The judge in this lesson is the function `judge` in `grade.py`, a few
rules written to act like a model judge. Rules give the same output on
every machine and every run, and a model call wouldn't. The function
gets the inputs a model judge gets, and it scores the two criteria in
`rubric.txt`:

```text
Complete (correctness)
  2  Answers every part of the question, as the reference answer lists
     the parts. When the handbook has no answer, says exactly: not found.
  1  Answers at least one part of the question, and misses at least one.
  0  Answers no part of the question.

Quoted (adherence)
  2  Quotes a sentence from the result of this run, word for word, or
     answers exactly: not found.
  1  Says in other words that the handbook has no answer.
  0  Answers without a quoted sentence from the result.
```

The file `pair.txt` has one question with two parts, how many days of
leave and until when they carry over, and two answers to it. Answer A
has two sentences. Answer B opens with a thank-you, says the same thing
at more length and closes with a wish. Both give the 25 days, both
quote the same sentence, and neither gives the carry-over date. A
person scores both 1 on Complete and 2 on Quoted.

#### Checkpoint: Which answer does the judge prefer?

Predict the scores the judge gives each answer, and which one it
prefers. The output repeats the question, gives each answer's length in
words and its two scores, and names the answer the judge prefers. Then
run it.

```sh
python3 grade.py prefer
```

The judge gives the long answer a 2 on Complete, and it misses the same
part of the question as the short one. That's the bias built into this
judge: an answer over 40 words gets one extra point on Complete, up to
the top of the scale. It's the constant `LENGTH_BONUS_WORDS` in
`grade.py`. A real model judge has no such line to read, but it shows
the same kind of bias. Model judges tend to prefer longer answers, and
to prefer the answer shown first (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). Answer B came second, so
the order didn't decide it here. In this fixture length is the whole
bias, and in a real judge you find a bias by testing for it, the way
the next section does.

## Compare the judge with a person

Before a judge grades at scale, it grades a sample that a person has
also graded, and you compare the two (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). A person graded all
twelve development runs with the same rubric, and the scores are in
`grades.csv`. The `agreement` step runs the judge on those runs and
counts the scores that match, per criterion.

#### Example: The judge against the person

Run this, and compare what you see with the output below.

```sh
python3 grade.py agreement
```

Output:

```text
Complete: judge and person agree on 9 of 12
  r03: person 1, judge 2 (67 words)
  r07: person 1, judge 2 (63 words)
  r11: person 0, judge 1 (58 words)
Quoted: judge and person agree on 12 of 12
```

All three disagreements are on one criterion. On Quoted the judge agrees
with the person on every run, because checking whether a sentence
appears word for word in the passage doesn't depend on how long the
answer is. On Complete it differs three times, each time upward and each
time on an answer of more than 40 words. Run three gives the meal limit
and leaves out the claim deadline, in a friendly paragraph. The search
in run eleven returned the laptops page, and the answer doesn't name an
owner, but it gets a 1 for its length. Runs five and nine are long too,
and on those two the judge and the person agree, because both answers
cover every part of the question. Length raises the score of an answer
that misses a part, and an answer that misses nothing is already at 2.

The fix goes where the disagreement is. The person applied the rubric
without trouble, so the rubric is clear, and the judge is what needs the
fix. For a model judge, that's the Complete criterion in the judge's
instructions: say that length and politeness don't add points, have the
judge list each part of the reference answer and mark it found or
missing before it scores, and then run the same sample again. Keep the
Quoted criterion as it is, because it agrees on every run. The rubrics
lesson set the bar: the judge is ready for a criterion when it agrees
with the person about as often as two people agree with each other.
Model judges can also miss a wrong fact they have no knowledge to check
(Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course), and the reference answer is what gives them that knowledge
here.

#### Checkpoint: What do you change before the judge grades at scale?

The judge and the person disagree on three of twelve runs, all on
Complete, all upward and all on long answers. What do you change first?

- Give both criteria of the judge a word limit, so no answer over 50 words scores 2.
- Replace the judge with a bigger model, because a judge that misses on 3 of 12 runs is too weak.
- Rewrite the Complete instructions so the judge checks each part of the reference answer, then rerun the sample.
- Average the judge and the person on the three runs, and let the judge grade the rest at scale.

## Numbers that rise while the agent stays the same

A team that tracks a number starts to change the agent until the number
goes up. If the number can go up while the quality stays the same,
that's often how it goes up, even when nobody means it to. So pick
**metrics that cannot be gamed** in that way: each one moves only when
the quality it stands for moves. Answer length, keyword presence and the
agent's own report of success are easy to raise. A test the agent can't
see, a person's spot check and an outcome measured later for the user
are harder.

A teammate adds a line to the agent's prompt that asks for full answers
that close with an offer of more help. The fixture adds the same closing
sentence to each development answer in code, and the rest of each answer
is unchanged. The sentence names the question's topic, and an answer of
`not found` doesn't get one. The `metrics` step prints four numbers for
the development runs, before and after that change. Keyword presence is
the share of answers that name the question's topic word, from
`reference.csv`.

#### Example: Four numbers, before and after

Run this, and compare what you see with the output below.

```sh
python3 grade.py metrics
```

Output:

```text
before      after
mean answer length  33 words    63 words
keyword present     7 of 12     11 of 12
judge, Complete     21 of 24    23 of 24
person, Complete    18 of 24    18 of 24
```

No answer gained a fact, and three of the four numbers went up. The
mean length nearly doubled because every answer except the `not found`
one got a closing sentence. Keyword presence went from 7 to 11 of 12,
because the closing sentence names the topic whether the answer is
right or wrong. Run eleven counted as a hit even before the change: it
names the payments team and not its owner.

The judge's Complete score rose from 21 to 23, because two short
answers that miss a part are now over 40 words. The fixture keeps the
person's grades, because the closing sentence adds no part of the
reference answer, and a person who reads for the parts of the question
gives the same scores.

Each easy number has a partner that measures the quality it was meant
to show. Length stands for completeness, so pair it with the parts of
the reference answer that the answer states. Keyword presence stands
for an answer from the right page, so pair it with a check in the trace
that the search returned that page. The judge's score stands for what a
person would give, so pair it with a person's scores on a fresh sample
every so often. When one of a pair moves and its partner doesn't, look
at the runs before you report either.

#### Checkpoint: Which partner catches each number?

Match each number with the partner that measures the quality the team reads it as.

Statements:

- The golden set pass rate after a month of prompt changes made while watching those items, read as how well the agent answers questions it hasn't seen
- The model judge's score after answers got longer, when the judge favors long answers, read as the score a person would give
- The share of conversations the agent marks as resolved, read as a sign that the user's problem went away

Options:

- The pass rate on held-out items that nobody tunes against
- A person scoring the same runs with the same rubric
- The share of users who don't ask the same question again within a week

## Check the change on the held-out runs

A number that measures the right quality can still mislead when the
team tuned against the same runs. After reading the development runs,
a teammate adds this line to the prompt, in `tuned/prompt-line.txt`:

```text
When a question has two parts, answer both. For remote work, give the
days and the allowance. For meals, give the limit and the claim
deadline. For training, give the budget and whether conference tickets
count. For onboarding, give the accounts and how long the checklist
takes.
```

The four examples are the four two-part development runs that answered
one part only. The author wrote their new answers, which are in
`tuned/`, and the person scored them. The other runs, the held-out ones
included, answer as before. The `held_out` step adds up the person's
Complete scores for each part of the set, before and after the change.

#### Example: Development and held out

Run this, and compare what you see with the output below.

```sh
python3 grade.py held_out
```

Output:

```text
Complete, scored by a person, before and after the tuned prompt
development: 18 -> 22 of 24
held out: 6 -> 6 of 8
```

The development score went from 18 to 22 of 24, and the held-out score
stayed at 6 of 8. Held-out runs h01 and h02 are two-part questions too,
about opening hours and about laptops, and the new line has no example
for them, so they still miss a part. The line fixed the four runs it
names. Report the held-out result as the effect of the change, and write
a line that asks for both parts without naming topics. Four held-out
runs is a small sample. A real set holds back about one item in five of
many more items, and the same comparison then has more runs behind it.

#### Checkpoint: What does the team report?

After the tuned line, the development part rose and the held-out part
didn't move. What does the team report as the effect of the change?

- No gain on unseen questions: held out stayed at 6 of 8, so the line fits the four topics it names.
- A gain of four points: development rose from 18 to 22 of 24, and that part has three times the runs.
- A gain of four points in 32: add both parts, 24 before and 28 after, to use every run in the set.
- Nothing yet: four held-out runs are too few, so the team reports once the set has a hundred.

#### Pitfall: A calibrated judge on a new model

A team compares its judge with a person on 30 runs, finds that they
agree on 27, and grades every night from then on. Half a year later the
vendor retires the judge's model, and the team moves the judge to the
newer one with the same instructions. The nightly score goes up, and the
team reads it as progress. The new model has its own biases, and the
calibration was a measurement of the old one. A judge's scores need a
comparison with a person's before anyone trusts them (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). This
course repeats that comparison each time the model behind the judge
changes: grade the same sample again, and compare it with the person
before you trust a single nightly score.

## Exercise

Copy the `llm-as-judge` folder and look at three numbers on its
development runs: mean answer length, keyword presence (the answer
names the topic word in `reference.csv`), and the judge's Complete
score. For each one, describe a change to the agent's replies that
would raise it while the agent answers no better, using a run from
`runs/` as the example. Then pair each number with a second number
that the same change doesn't raise. Naming the change for each number
makes the way it can be gamed concrete, before a real change raises
it.

A good result names a concrete change for each number, such as a
closing sentence or a restated question, and shows it on a run. Each
partner measures the quality the first number was meant to show, and
you can say why the change doesn't raise it. Compare your answer with
`model-answer.txt` in the folder. Which of your three partners costs
the most to measure, and how often would you measure it?

Stretch: Pick one number your own team tracks about an agent or a tool, write down one way it could rise with nothing getting better, and name its partner.

## Recap

1. A model judge gets the question, the answer, the criteria and often
   a reference answer, and scores runs at a scale no person can read
   (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course).
2. Model judges tend to prefer longer answers and the answer shown
   first (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). Test for those biases with pairs that differ only
   in length or order.
3. Before a judge grades at scale, compare its scores with a person's
   on a sample, per criterion, and fix the criterion where they
   disagree (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course).
4. Length, keyword presence and self-reported success can rise with
   the agent unchanged. Pair each number with one that measures the
   quality it stands for.
5. Report a change by its effect on held-out runs, and grade the sample
   again whenever the model behind the judge changes.

#### Habit

The next time a number about an agent or a tool goes up, name one way it could have risen with nothing getting better before you report it.

## References

- Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/09-evaluating-and-testing-agents/