# Seeing bias across many answers

> Read two shortlists from a simulated ranker on the page that each look fair on their own, count who made the cut when the names are swapped, and decide how much checking a task that affects people gets.

Page: https://schubergphilis.github.io/ai-training/safety/bias-in-patterns/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson we read two shortlists that a ranker drew from the same
twelve candidate records, and each shortlist looks fair on its own. Then we
count who made the cut across both, and the count says something neither
shortlist says by itself. By the end you know why bias in a model's output
is a pattern you find by sampling and comparing, and how much checking a
task gets when its output affects people.

You need only this page. The ranker in this lesson is a simulation, a
small program written for the lesson that runs in your browser, and the
records and names are made up. Its numbers say what a slant looks like in a
table of results, and they are not evidence about any real model. The real
evidence is in the papers cited in the second section.

## Each shortlist looks fine

A team has twelve candidate records for one role, and each record has the
same fields: years of experience, number of certificates, and months since
the last role. The team gives the records to a ranker, which scores each one
and shortlists the top six. Here is the shortlist it returned.

> 1. Arvel Dunmoor (experience 7, certificates 2, gap 3)
> 2. Fenwick Sabani (experience 7, certificates 2, gap 4)
> 3. Pell Varnava (experience 6, certificates 1, gap 2)
> 4. Brisa Okonde (experience 6, certificates 0, gap 3)
> 5. Nadir Kesrou (experience 5, certificates 1, gap 1)
> 6. Corin Halvane (experience 5, certificates 0, gap 1)

Read it as the team would. The most experienced people are at the top, a
long gap costs a place, and a certificate lifts a record over one with the
same years. You would sign off on this list.

Now the same twelve records, ranked again, and this time each record
has a different name. The names come from two lists of six, list A and
list B. Both lists are invented for this lesson, and first names and
surnames are mixed across them on purpose, so that neither list reads as
one group. In the first run half the records had list A names and half had
list B names. In the second run the lists are swapped, so every record has
now had one name from each list.

> 1. Marisel Adair (experience 7, certificates 2, gap 3)
> 2. Rashon Belka (experience 7, certificates 2, gap 4)
> 3. Dessa Tesfay (experience 6, certificates 1, gap 2)
> 4. Brisa Okonde (experience 5, certificates 1, gap 1)
> 5. Fenwick Sabani (experience 4, certificates 2, gap 1)
> 6. Elior Vance (experience 4, certificates 1, gap 0)

This one reads fine too. The same three records lead, and the last places
went to shorter records with no gap, which is a defensible call. List A is
Arvel Dunmoor, Brisa Okonde, Corin Halvane, Dessa Tesfay, Elior Vance, and
Fenwick Sabani. List B is Marisel Adair, Nadir Kesrou, Oriel Morrow, Pell
Varnava, Quilla Tallent, and Rashon Belka. Before reading on, count how
many places on each shortlist went to a list A name.

## Count who made the cut

The ranker is a widget further down this page. It scores each record from
its three fields and shortlists the six best. Then it swaps the name lists
and does it again. Its last line says how many shortlist places went to each
list, added up over both runs. When two records tie at the shortlist line,
the record that the team listed first gets the place.

The swap is the point of the design. A ranker that reads only the fields
shortlists the same six records in both runs, because the fields did not
change. Each of those six records had a list A name in one run and a list B
name in the other. So the totals over both runs come out equal, six and six,
whatever the records say. If the totals are not equal, the name changed the
score, and the difference is the size of the slant.

#### Checkpoint: What are the totals?

Before you press the button, predict the last line the ranker prints.

- shortlisted over both runs: list A 8, list B 4
- shortlisted over both runs: list A 6, list B 6
- shortlisted over both runs: list A 4, list B 2
- shortlisted over both runs: list A 4, list B 8

Now press the button, with the slider at 2 points, where it starts, and
compare the last line with your prediction.

Eight and four. You could not have read that from one shortlist. In each
run list A took four of six places, and a reader who saw either one saw a
plausible split, because a 4-2 split one way or the other is common by
chance, and one run proves nothing. Only the pair, with the names swapped
and the counts added, shows that in both runs a place went with the name.
The widget also prints both rankings with their scores. At 2 points it
prints this:

```text
Run 1
   1. shortlist  score 24  Arvel Dunmoor   list A  (experience 7, certificates 2, gap 3)
   2. shortlist  score 23  Fenwick Sabani  list A  (experience 7, certificates 2, gap 4)
   3. shortlist  score 18  Pell Varnava    list B  (experience 6, certificates 1, gap 2)
   4. shortlist  score 17  Brisa Okonde    list A  (experience 6, certificates 0, gap 3)
   5. shortlist  score 16  Nadir Kesrou    list B  (experience 5, certificates 1, gap 1)
   6. shortlist  score 16  Corin Halvane   list A  (experience 5, certificates 0, gap 1)
   7.            score 15  Rashon Belka    list B  (experience 4, certificates 2, gap 1)
   8.            score 14  Quilla Tallent  list B  (experience 4, certificates 1, gap 0)
   9.            score 13  Dessa Tesfay    list A  (experience 4, certificates 1, gap 3)
  10.            score 11  Oriel Morrow    list B  (experience 3, certificates 2, gap 2)
  11.            score 10  Marisel Adair   list B  (experience 4, certificates 1, gap 4)
  12.            score  9  Elior Vance     list A  (experience 2, certificates 2, gap 3)
  shortlisted: list A 4, list B 2

Run 2: the same records, names swapped
   1. shortlist  score 22  Marisel Adair   list B  (experience 7, certificates 2, gap 3)
   2. shortlist  score 21  Rashon Belka    list B  (experience 7, certificates 2, gap 4)
   3. shortlist  score 20  Dessa Tesfay    list A  (experience 6, certificates 1, gap 2)
   4. shortlist  score 18  Brisa Okonde    list A  (experience 5, certificates 1, gap 1)
   5. shortlist  score 17  Fenwick Sabani  list A  (experience 4, certificates 2, gap 1)
   6. shortlist  score 16  Elior Vance     list A  (experience 4, certificates 1, gap 0)
   7.            score 15  Nadir Kesrou    list B  (experience 6, certificates 0, gap 3)
   8.            score 14  Oriel Morrow    list B  (experience 5, certificates 0, gap 1)
   9.            score 13  Corin Halvane   list A  (experience 3, certificates 2, gap 2)
  10.            score 12  Arvel Dunmoor   list A  (experience 4, certificates 1, gap 4)
  11.            score 11  Pell Varnava    list B  (experience 4, certificates 1, gap 3)
  12.            score  7  Quilla Tallent  list B  (experience 2, certificates 2, gap 3)
  shortlisted: list A 4, list B 2

shortlisted over both runs: list A 8, list B 4
```

Look at places six and seven in each run. In the first run, Corin Halvane
from list A is shortlisted with a score of 16, and Rashon Belka from list B
is out with 15. On the fields alone Rashon Belka's record scores 15 and
Corin Halvane's scores 14. In the second run, Elior Vance from list A is in
with 16 and Nadir Kesrou from list B is out with 15, and on the fields alone
Nadir Kesrou's record is the stronger one by a point. At 2 points the
simulation adds two points to every list A name, and in each run those two
points moved one record across the shortlist line. One record per run is a
difference of four in the totals. A real model has no such setting that you
can look up or change. The effect is spread through its weights, and this
simulation exists so that you can see what the effect does to a table of
results without reading any weights.

Move the slider to 0 and press the button again. With no points added, the
two runs shortlist the same six records, and the last line reads six and
six, as the design of the swap guarantees. Equal totals from a dozen
records still don't prove that the name changes nothing, because a small
slant may move no record across the line.

## Where the slant comes from

**Bias** in a model is a systematic slant in its output that it inherited
from its training data and its tuning. The model learned which words follow
which from a large body of text, and it predicts the most likely
continuation of whatever it is given
(AI capabilities and limitations, Claude Academy). In that text some names appear
more often next to some jobs, and a name that appears often in the text at
all reads differently from one that appears rarely. The same record reads as
a slightly better fit under one name than under another (Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7 (AIES 2024), 1578-1590).
Tuning after training can reduce a slant or add a new one, because the
people and the rubrics that grade the model's answers have preferences of
their own.

This is measured, and the measurement uses the same swap as the widget.
Bertrand and Mullainathan sent nearly five thousand fictitious resumes to
real job advertisements, identical except for the name, and found that
resumes with white-sounding names got about fifty percent more callbacks
than the same resumes with African-American-sounding names (Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination, American Economic Review 94(4) (2004), 991-1013).
Wilson and Caliskan ran the method against text-embedding models set up to
retrieve resumes for a job description, with over five hundred resumes and
job descriptions across nine occupations, and found that the models favored
white-associated names in 85.1 percent of the tests, favored
female-associated names in 11.1 percent, and in some settings put Black male
names at a disadvantage in every test (Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7 (AIES 2024), 1578-1590). The models were not
told anyone's race or gender. The name was enough, as it was for the
employers in 2004.

Both studies needed thousands of comparisons to find this. One output of
a slanted model is a plausible answer, the same as one output of a fair
one. The slant is a shift in the odds, and odds show up only when you
have many draws to count. That is what makes bias different from the
invented fact in the previous lesson: a hallucination can be caught in the
sentence it appears in, and a slant cannot.

#### Pitfall: It ranked one candidate the way I would have

A manager tests a screening assistant by giving it three applications she
knows well and asks it to rank them. The order matches hers and the reasons
read well, and she rolls it out for the next hundred. Each of the three
outputs was reasonable, and three outputs can't show a slant of two points
in a score, which is all it took to move a record across the shortlist line
above. Rule: a fairness check is a comparison of counts across groups over
many records, and one agreed answer, or three, is evidence about those
answers and nothing else.

## How much checking a screening task gets

The lesson [What an invented fact looks like](https://schubergphilis.github.io/ai-training/safety/spotting-hallucination/)
sized the checking to the stakes of a claim. The same rule applies here,
and the stakes of a task that affects people are set by who it affects. A
model that drafts your meeting notes affects you, and you catch its
mistakes as you read. When a model screens applications or triages support
tickets by customer, it affects people who never see the output and can't
correct it. That is the high-stakes end, and the checking it gets is
different in kind from reading a few outputs.

The check that fits is the one the widget did. Name the groups the task
could treat differently. Take a sample of records that is large enough for a
pattern to show, and the widget shows that a dozen is barely enough for a
slant of two points. Run the same records under swapped markers of the
group, such as a name or a postcode, and count the outcomes per group. Give
the counts to someone who can judge the records and who has the standing to
stop the rollout. The team trusts the tool for that task after the
comparison comes back level, and keeps repeating the comparison, because the
model changes and the applicants change.

#### Checkpoint: The internship shortlist

A team wants to use a chat assistant to shortlist twenty interview
candidates from three hundred internship applications. Time is short. What
checking does the task get before the team trusts the tool for it?

- Ask the assistant to explain its reasons for each shortlisted applicant, read the reasons, and go ahead if none of them names a group.
- Have a recruiter spot-check five of the twenty, and go ahead if those five look like people the recruiter would have picked.
- Run a sample with names swapped across the groups the team can name, count the shortlist rate per group, and have the recruiting lead decide on the counts.
- Rely on the vendor statement that the model was tested for fairness before release, and keep a copy of the statement on file.

## Exercise

Pick one task at your work where a model's output affects people who don't
see it: a shortlist, a triage queue, a scoring, a decision about who gets
contacted first. Write down which groups the output could treat differently
and what marker in the input signals each group. Then write down how many
records you would need to compare before you would trust a pattern, and
who reads the counts and has the standing to stop the rollout. Fifteen
minutes is enough.

A good result names at least two groups, one input marker per group, a
sample size with a reason attached, and a named person or role who decides.
Is that person the one who wants the tool, or someone who can say no to
it?

Stretch: Set the slider in the widget to 3 and then to 4, and press the button each time. Which of them give the same totals as 2 points? What does that tell you about how precisely a dozen records measure a slant, and how many records a check needs?

## Recap

1. Bias is a systematic slant a model inherited from its training data and
   tuning. It shifts the odds, and one output of a slanted model looks
   as reasonable as one output of a fair one (Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7 (AIES 2024), 1578-1590).
2. The slant becomes visible in totals: the same records under swapped
   group markers, the outcomes counted per group, compared. A fair ranker
   gives equal totals by construction, so a difference is the slant.
3. This has been measured in employers and in models with the same
   swapped-name method, and the models favored some names for the same
   record without being told anything about the person (Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination, American Economic Review 94(4) (2004), 991-1013)
   (Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7 (AIES 2024), 1578-1590).
4. A task whose output affects people who can't correct it is high stakes,
   and its check is a sampled comparison across groups, read by someone who
   can judge the records and stop the rollout, repeated as the model and
   the inputs change.

## References

- AI capabilities and limitations, Claude Academy, https://academy.claude.com/courses/ai-capabilities-and-limitations
- Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7 (AIES 2024), 1578-1590, https://doi.org/10.1609/aies.v7i1.31748
- Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination, American Economic Review 94(4) (2004), 991-1013, https://www.aeaweb.org/articles?id=10.1257/0002828042002561