# When the tool is usually right

> Why attention to a tool's output decays fastest when the tool is good, what aviation and medicine found out about that, and how to build the check into the process rather than into willpower.

Page: https://schubergphilis.github.io/ai-training/safety/when-the-tool-is-usually-right/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson we look at what happens to a person's attention when a
tool has been right for a long time, and why the rare wrong answer then
passes. The earlier lessons in this part showed an invented fact and a
slant across many answers. This one is about the reader, because a failure
the reader has stopped looking for is one nobody catches. By the end you
can name the effect and set up a check that still works on a Friday
afternoon in month six, when willpower has run out.

Consider a team that uses a chat assistant to draft the covering note for
every invoice it sends. The team is invented for this lesson. In the first
week everyone reads each note before it goes out and fixes a wrong date or
a misread total. In the first month they find two mistakes in about two
hundred notes. By the second month nobody reads the notes any more. The
assistant is good at this, the notes all look the same, and reading them
feels like a waste of a skilled person's time. In the fifth month a
customer replies to ask why the note offers a discount for early payment
that the invoice does not mention. The assistant had picked the sentence
up from an earlier customer's terms, and it had been in every note to
that customer for three weeks. Everyone on the team had been careful in
the first month. The process had lost its check since then, and the tool
being good is what removed it.

## Trust that spreads

**Overreliance** is trusting a tool's output beyond what you have checked
or could check (AI capabilities and limitations, Claude Academy). It starts in a
place that looks safe. You know the task well. The first fifty outputs
were right when you read them closely, so you stop reading closely. The
trust is built on that task. Then the same tool is used on a task next to
it, where you know less, and the trust carries over without anyone
deciding that it should. On the first task you could tell a good answer
from a plausible one at a glance. On the second you can't, and a model
produces plausible text whether or not it is right, because fluent text is
what it is built to produce. The spread is the problem. Trust that was
calibrated on tasks you could judge ends up covering tasks you can't.

The invoicing team shows the pattern in one setting. The notes were a task
the team knew. The customer's payment terms were a fact stored in a
different document, and no one on the team was reading the notes against
it, so the tool's mistake about the terms looked like every other correct
note. The check that would have caught it was a comparison, and a comparison
is exactly the kind of reading people stop doing once the output looks
right every time.

## A known effect with a name

Researchers who study people working with automated systems have a name
for the second half of this story. **Automation complacency** is the
habit of waving a tool's output through because it has been right many
times before. Parasuraman and Manzey reviewed decades of studies on
pilots, operators and clinicians and describe complacency and automation
bias as effects of how attention is spent. When a person has other
tasks to attend to and one of them is a reliable machine, attention goes
to the others, and the machine's rare failure arrives when nobody is
watching it (Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors 52(3) (2010), 381-410). The reliability is the cause. A tool
that is wrong one time in three keeps people reading. When the tool is wrong
one time in three hundred, it trains them to stop.

Aviation is where the effect was first studied in depth, and the accident
reports name it. In 2013 an airliner on approach to San Francisco struck
the seawall short of the runway. The National Transportation Safety Board
found that the crew had not noticed that the autothrottle was no longer
controlling the airspeed. Among its findings it gave the crew's reliance
on the automation as one reason the airspeed went unmonitored, and it
listed the complexity of the autothrottle and autopilot systems among the
contributing factors (Descent Below Visual Glidepath and Impact With Seawall, Asiana Airlines Flight 214, Boeing 777-200ER, HL7742, San Francisco, California, July 6, 2013 (AAR-14/01), National Transportation Safety Board). The crew were trained
professionals in a cockpit built for monitoring. Experience with the
system did not protect them from the mode it was in.

Medicine has the same finding for decision support software, the systems
that suggest a diagnosis or flag a drug interaction. Goddard, Roudsari and
Wyatt reviewed the studies that gave clinicians deliberately wrong advice
and found that clinicians followed a wrong suggestion more often than a
control group that worked without the system. In the studies they reviewed, a
correct decision was changed to a wrong one on the system's advice in
between 6 and 11 percent of cases. The effect was larger under
time pressure and in people less sure of their own answer, and smaller in
the experienced (Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association 19(1) (2012), 121-127). The clinicians in these studies were
making a normal trade of attention against a tool with a good record, and
laziness has no part in it.

#### Pitfall: I need to concentrate harder

A team lead reads the story of the invoicing team and resolves that from
now on she will read every note the assistant produces with full
attention. For two weeks she does, and finds nothing, and in the third
week a long day pushes the notes to the end of the queue and she signs
them off in a batch. The resolve was sincere, and it was aimed at the
wrong place. Attention to a source of rare errors decays in everyone, and
the decay is faster when the person is busy and the tool is good
(Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors 52(3) (2010), 381-410). Rule: treat complacency as a property of the setup
you are in, and change the setup. A person who has decided to be more
careful has changed nothing that will still be true in month six.

## Build the check in

If attention cannot be relied on, the check has to live somewhere else.
The counters that hold up are all process. They work when the person
running them is tired, and they work for the new colleague who never saw
the tool make a mistake. These are the ones that fit most settings.

- **A designed check.** Decide, before the tool is trusted for a task,
  what a wrong output would cost and who would see it, and write down
  what gets checked and against what. For the invoicing team that would
  have been "the note is compared against the invoice and the customer's
  terms". A check that has a name and a place in the process is still done in a
  busy week. A check that exists only in someone's intention is skipped.
- **Sampling.** Reading everything is not sustainable, and it is not
  what a check needs. Pick a fixed share of outputs, or every output above
  a stake threshold, and read those with the source open. The share can be
  small when the stakes are low. It is fixed in advance rather than left
  to whoever has time.
- **Rotating who reviews.** One reviewer's attention decays on the
  same curve as everyone else's. When the review moves between people, the
  fresh eyes arrive on a schedule, and a mistake that has been there for
  three weeks looks new to someone. The medical review found that giving
  people the reasons behind a suggestion rather than a bare answer reduced
  how often they followed wrong advice, and that making them accountable
  for the decision helped in some studies and not in others
  (Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association 19(1) (2012), 121-127). Both are process changes, and neither asks anyone to
  try harder.
- **Keeping your own skill.** Bainbridge pointed out in 1983 that an
  operator who only watches a machine loses the skill to do the task, and
  is then asked to take over exactly when the machine fails
  (Ironies of automation, Automatica 19(6) (1983), 775-779). A person who hasn't drafted a covering note or
  checked a figure for a year is in the same position. Do part of the task by hand
  now and then, so that you can still tell a wrong output when you see
  one.

Match the depth of these to the stakes, as the earlier lessons in this
part did for an invented fact and for bias. An internal brainstorm list
needs none of them. Output that goes to a customer or can't be taken
back gets a designed check and a named reviewer, and so does an output in
a domain you can't judge. The tool's good record is the reason the check
has to be designed, because that record is what removes the reading.

#### Checkpoint: Three months without a mistake

A support team has used a chat assistant to draft replies for three
months. Nobody has found a mistake in the drafts since the first two
weeks, and the replies go out under the team's name. What checking does
the task get from here?

- Keep reading every draft, and remind the team at the next meeting that the drafts still need a careful read.
- Stop checking. Three months without a mistake is enough evidence that the tool is reliable for this task.
- Set a fixed sample, say one draft in ten plus every reply about money or a cancellation, read those with the ticket open, and rotate the reviewer each week.
- Ask the assistant to add a line at the end of each draft saying whether it is confident, and read only the drafts it marks as uncertain.

#### Checkpoint: Which counter is process?

A team proposes four ways to stop waving the assistant's drafts through.
Which one is a change to the process rather than a request for more
willpower?

- Everyone on the team commits to reading the drafts with the same care they gave them in the first week.
- The team lead puts a note above her desk that says "the tool can be wrong".
- Every draft that mentions a price or a date is compared against the invoice by whoever is on review duty that week.
- New team members are told the story of the discount sentence on their first day, so that they know the tool has failed before.

## Exercise

Pick one task at your work where you or your team once read a tool's
output closely and no longer do. Write down what the last unchecked
mistake would have cost and who would have been the first to find it.
Then write down what a designed check for it would compare against what.
Ten minutes is enough. The exercise shows that the tasks you have stopped
checking are the ones where the tool has your trust, and that this is the
condition under which a mistake passes.

A good result names a specific cost (a refund, a wrong figure in a report
that went to a client, an hour of a colleague's time), a specific finder,
and a check with a size and an owner. If the finder you wrote down is
outside your team, does the depth of your check match that?

Stretch: Do the same for a task a colleague has stopped checking, and compare the two answers on who would find the mistake. If the answer is 'the customer' for either, that is the first check to design.

## Recap

1. Overreliance is trusting a tool's output beyond what you have checked
   or could check. It starts on tasks you know well and spreads to tasks
   where a good answer and a plausible one look the same
   (AI capabilities and limitations, Claude Academy).
2. Automation complacency is the habit of waving a usually-right tool's
   output through. It is an effect of how attention is spent, found in
   trained pilots and clinicians, and it is stronger when the tool is
   reliable and the person is busy (Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors 52(3) (2010), 381-410) (Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association 19(1) (2012), 121-127).
3. Willpower is the wrong counter, because attention decays in everyone.
   The counters that hold are process: a designed check, a fixed sample,
   a rotating reviewer, and doing part of the task by hand so that you can
   still see a wrong output (Ironies of automation, Automatica 19(6) (1983), 775-779).
4. Attention decays fastest when the tool is good, so the check has to be
   built in before the tool has the record that would remove it.

## References

- AI capabilities and limitations, Claude Academy, https://academy.claude.com/courses/ai-capabilities-and-limitations
- Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors 52(3) (2010), 381-410, https://doi.org/10.1177/0018720810376055
- Descent Below Visual Glidepath and Impact With Seawall, Asiana Airlines Flight 214, Boeing 777-200ER, HL7742, San Francisco, California, July 6, 2013 (AAR-14/01), National Transportation Safety Board, https://www.ntsb.gov/investigations/AccidentReports/Reports/AAR1401.pdf
- Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association 19(1) (2012), 121-127, https://doi.org/10.1136/amiajnl-2011-000089
- Ironies of automation, Automatica 19(6) (1983), 775-779, https://doi.org/10.1016/0005-1098(83)90046-8