# Assessing the risk of a use case before it starts

> Rate three proposed AI use cases on impact and likelihood, find the ones at or above the threshold, and decide between extra controls and a written acceptance, with a record that can be reviewed.

Page: https://schubergphilis.github.io/ai-training/safety/assessing-a-use-case/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In one week, a team lead gets a proposal for an assistant that listens to
the team's meetings and writes the summary. The next day a second proposal
arrives, for an assistant that drafts the internal newsletter from the
updates people send in. On Thursday a third asks for an agent that reads
the customer mailbox and answers the mail itself. Each proposal comes with
a demo that went well, and each person asks for a yes by Friday.

The team's [AI policy](https://schubergphilis.github.io/ai-training/safety/writing-an-ai-policy/) says which tasks an
agent may do. It doesn't say which new use case is safe to start, and a
demo doesn't answer that question. In this lesson we answer it with a
**risk assessment**: a short written method that rates each proposal
before it starts and says which ones need a decision from someone with the
authority to take the risk. You need a document or a chat assistant with no
tools, and about fifteen minutes.

## The method on one page

Write down the result of each step.

1. Name the domains the use case touches. Personal data, money, safety
   (the physical safety of people) and reputation cover most cases. Next
   to the domains, name the people who are affected when it goes wrong.
2. Rate the impact of the worst realistic mistake, and the likelihood that
   it happens, on scales the organization agreed in advance.
3. Combine the two ratings into a level.
4. Compare the level with the agreed threshold. Below it, the team goes
   ahead under its policy. At or above it, the use case needs extra
   controls, or a written acceptance from the person who answers for the
   risk.

Here are the scales this lesson uses. An organization may pick other words
and other numbers. Everyone must rate on the same scales, so
two ratings can be compared.

```text
Impact of the worst realistic mistake
  1 minor        one person is inconvenienced, and it is fixed within a day
  2 moderate     a few people are affected, or a small cost, fixed within a week
  3 major        many people are affected, or personal data reaches people who
                 should not see it
  4 severe       customers or the public are harmed, a legal duty is broken,
                 or the loss can't be undone

Likelihood that it happens
  1 rare             not expected in a year of use
  2 possible         a few times a year
  3 likely           about once a month
  4 almost certain   every week

Level = impact x likelihood
  1 to 3 low    4 to 6 medium    8 to 9 high    12 to 16 very high

Threshold: 8. A level of 8 or more needs extra controls or a written acceptance.
```

The first step comes first on purpose. A rating made before the domains
are named is a rating of the demo. The domains and the affected people are
the blast radius of the use case: what a wrong step can reach, and who
feels it. They decide how bad the worst realistic mistake is, and so they
decide the impact rating.

How easily a mistake can be undone is part of its impact, which is why the
impact scale says "fixed within a day" and "can't be undone". Sort the work
by how easily it can be undone (Guardrails and safety: why agent safety differs, defense layers, injection, human in the loop, Agent Engineer Course). A sent email or a payment is
final, and a draft nobody has read yet can still be thrown away.

The likelihood rating comes from the real volume of work and from how
quickly the team would notice a mistake. A mistake caught in a
draft never reaches a customer, and one that nobody notices for a month
has a month to happen again.

#### Checkpoint: Where does the assessment start?

A hiring manager proposes an assistant that sorts incoming job
applications into "interview" and "no". You are asked to assess it before
it starts. What do you write down first?

- Rate the likelihood from the pilot results, since a pilot shows how often the tool gets it wrong.
- Write down the domains it touches and the people it affects: the applicants, their personal data, and the hiring managers who act on the sort.
- Ask the vendor for the accuracy figure, and use it for both the impact and the likelihood ratings.
- Start the tool on a few real applications, and rate it once the team has seen what it does in daily use.

## Scoring the three proposals

The team lead writes down the domains, the people affected and the two
ratings for each proposal.

- **Meeting summarizer.** It runs in team meetings and in one-to-ones,
  where health and performance come up. The summaries are saved in the
  shared team folder. Domains: personal data, reputation. Impact 3, because
  personal data reaches people who should not see it. Likelihood 3, because
  the one-to-ones are weekly and the folder is shared by default.
- **Newsletter drafter.** An editor reads every draft before it goes out to
  the staff. Domain: reputation. Impact 2, likelihood 2.
- **Customer mail agent.** It reads the customer mailbox and sends its
  replies without a person reading them. Customers write about orders,
  refunds and their accounts. Domains: personal data, money, reputation.
  Impact 4, because a wrong reply can promise a refund or show one
  customer another customer's details. Likelihood 3, because the mailbox
  gets thousands of messages a month.

The team lead can say yes alone only to a proposal below the threshold.
A proposal at or above it needs extra controls, or a written acceptance
from the person who answers for the risk. A small program applies the
method to these ratings and prints the level of each proposal and whether
it is at or above the threshold. Before you read its output, work out the
three answers yourself.

#### Checkpoint: Which proposals can't the team lead approve alone?

Which proposals need extra controls or the risk owner's written acceptance
before they start? The program prints one `at or above threshold:` line
per proposal, in the order above, and a `yes` means the team lead can't
approve that proposal alone. Type the three values, `yes` or `no`,
separated by spaces.

The program's output:

```text
threshold: 8 (high)

use case: meeting summarizer
  domains: personal data, reputation
  affected: everyone in the meetings, and anyone discussed in a one-to-one
  impact: 3 (major)
  likelihood: 3 (likely)
  level: 9 (high)
  at or above threshold: yes

use case: newsletter drafter
  domains: reputation
  affected: the staff who read the newsletter
  impact: 2 (moderate)
  likelihood: 2 (possible)
  level: 4 (medium)
  at or above threshold: no

use case: customer mail agent
  domains: personal data, money, reputation
  affected: every customer who writes in, and the support team
  impact: 4 (severe)
  likelihood: 3 (likely)
  level: 12 (very high)
  at or above threshold: yes
```

The customer mail agent was the expected result. The meeting summarizer is
the one that surprises people, because summarizing a meeting sounds
harmless. The domains changed the answer. Once the list says "personal
data" and "anyone discussed in a one-to-one", the summarizer is a tool that
copies health and performance remarks into a shared folder every week.

The newsletter drafter goes ahead under the team's policy. The other two
wait for the decision in the next section.

#### Pitfall: Rating likelihood from the demo

The customer mail agent's demo answered twenty test messages, and every
reply was right. The person proposing it rates the likelihood 1, rare.
Rated that way, the level is 4 (medium), and the agent is below the
threshold.
The twenty messages were picked for the demo, and the real mailbox gets
thousands a month, with refunds, complaints and messages in other
languages among them. A demo shows that the tool can do the task. It
doesn't show how often it fails on the messages nobody picked. Rate
likelihood from the volume and the variety of the real work, and from how
quickly a mistake would be noticed.

## At or above the threshold

A use case at or above the threshold doesn't stop there. It gets one of two
decisions, and both are written down.

The first is **extra controls** that bring the level down. The team lead
changes the customer mail agent so that it drafts each reply and a person
from the support team reads and sends it. The impact of a wrong reply is
the same as before, but a wrong draft is now caught before a customer sees
it. Rated again, the likelihood is 1 (rare) and the level is 4 (medium),
below the threshold. Rare holds only while a person really reads each
draft. The support team already read and answered every message by hand,
so reading a draft is less work than they did before. If the drafts arrive
faster than people can read them, the approvals stop being read, and the
rating goes back up. The approval sits right before the step that can't be
undone, as the [human in the loop](https://schubergphilis.github.io/ai-training/safety/agent-risk/#human-in-the-loop)
section of the agent risk lesson describes. The rating is only as good as that control,
so the record names it.

The second is **acceptance in writing** by a person with the authority to
accept the risk. That is the person who answers for the domain when it goes
wrong: for the customer mailbox, the head of customer service. The person
who proposed the tool, or the team that wants it, can't accept the risk on
their own. A person with the authority may accept it for a limited time, or
with a review date, and signs a record that says so. An organization may
name a more senior person for the very high band, and the record then
carries that person's name.

For the meeting summarizer the team lead picks a control that shrinks the
list of affected people. The assistant stays out of one-to-ones, and each
summary goes only to the people who were in the meeting. The team lead
rates it again with the smaller list, and the new ratings go in the record.

Whichever decision is made, the record holds the same lines.

```text
Risk record: customer mail agent
Domains        personal data, money, reputation
Affected       every customer who writes in, and the support team
Ratings        impact 4 (severe), likelihood 3 (likely), level 12 (very high)
Decision       extra control: the agent drafts, a person reads and sends each reply
Rated again    impact 4 (severe), likelihood 1 (rare), level 4 (medium)
Decided by     head of customer service
Review         in three months, or after the first incident
```

The record matters later. When a wrong reply does reach a customer, the
team can read what was assumed, who decided and which control was supposed
to catch it, and fix the rating or the control. Without the record the
same question gets a new answer from whoever is asked. An auditor, or the
person who takes over the team next year, can review a decision only when
it is written down. The [EU AI Act lesson](https://schubergphilis.github.io/ai-training/safety/eu-ai-act/) comes next,
and it covers the uses that the law itself sorts into risk levels.

#### Checkpoint: Who signs the acceptance?

The team decides not to add controls to the customer mail agent for now,
and to ask for a written acceptance of its level instead. In this
organization, the person who answers for an area signs the acceptances for
it at every level. Who should sign?

- The support team member who proposed the agent, since they know best how it works.
- The vendor of the agent, because the vendor built it and knows its failure rate.
- The head of customer service, who answers for the mailbox and its customers.
- The whole support team together, by a vote at the next team meeting.

## Exercise

Assess one AI use case from your own work, one that is proposed or that
you already use, in a document or in a chat with an assistant that has no
tools. Write down the domains it touches and the people a mistake would
affect, rate the impact and the likelihood on the scales in this lesson,
work out the level and compare it with the threshold. If the level is 8 or
more, name one control that would bring it down, and the person who would
have to accept it in writing if you added none.

A good result fits in a record like the one above, and every
rating has a reason next to it that someone else could check. If a
likelihood reason mentions a demo or a trial run, rate it again from the
real volume of work. Which domain did you almost leave off the list?

Stretch: Show your assessment to a colleague and ask them to rate the impact and the likelihood without seeing your ratings. Where your numbers differ, write down which fact explains the gap.

## More practice

#### Checkpoint: Six months later

Six months after an agent started, a wrong reply reaches a customer. The
team looks for the assessment, and finds only a line in the meeting notes:
"agent approved". Nobody can say whether the rating was wrong or a control
was missing. Which change lets the team answer that question after the
next incident?

- Keep writing one line in the meeting notes each time a tool is approved, as the team does now.
- After each incident, rate the use case again from scratch, using what the team knows then.
- Stop approving agents until the team can prove that the assessment method works.
- When each use case starts, write a record with the ratings, the decision, the control, and who decided.

## Recap

1. A risk assessment starts with the domains a use case touches (personal data, money, safety and reputation) and the people a mistake affects. That list sets the impact rating.
2. Rate impact and likelihood on scales everyone agreed, multiply them into a level, and compare the level with the agreed threshold.
3. Sort the work by how easily a mistake can be undone (Guardrails and safety: why agent safety differs, defense layers, injection, human in the loop, Agent Engineer Course), and put that in the impact rating. Rate likelihood from the real volume of work and from how fast a mistake is noticed. A demo shows the tool can do the task, and says little about how often it fails.
4. At or above the threshold, add a control that brings the level down, or get a written acceptance from the person who answers for the risk.
5. The record is what makes the decision reviewable. It holds the domains, the ratings, the decision, the control and the name of who decided.

#### Habit

The next time someone proposes a new AI tool for your team, ask which domains it touches and who a mistake would affect before anyone talks about how well the demo went.

## References

- Guardrails and safety: why agent safety differs, defense layers, injection, human in the loop, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/10-guardrails-and-safety/