# Drafting a text and asking for critique

> Brief a chat assistant for a short email and check every fact in the draft, then ask for critique of it as a colleague's text against criteria you wrote first, and get problems in place of praise.

Page: https://schubergphilis.github.io/ai-training/using-agents/drafting-and-critique/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson we ask a chat assistant for a short email twice, first with
one line and then with a brief. Then we hand the result back and ask what
it thinks, and we get praise. The second time we give it criteria we
wrote before the draft existed, and the assistant finds the weak point.
By the end you have one brief for a draft and one for a critique.

You need any chat assistant you already have access to. Every exchange on
this page is illustrative. The author wrote it to show the pattern, and
your assistant gives different words.

## A one-line request

Suppose your team of twelve changes how it claims travel expenses. From
1 November everyone uses a new online form. You want to tell them in a
short email, and you ask for one.

#### Prompt (illustrative, not a recorded transcript)

```text
Write an email to my team about the new expense process.
```

#### Response

````text
```text
Subject: Important update to our expense process

Dear team,

I am writing to let you know about some upcoming changes to the way we
submit expenses. Starting next month, we will move to a new system that
makes submitting claims faster and easier for everyone.

The new process will simplify approvals and reduce paperwork. Please
review the attached guidelines before you submit your next claim, and
contact [Name] in Finance if you have any questions.

Thank you for your cooperation.

Best regards,
[Your name]
```
````

The email reads well, and you can't send it. It doesn't give a date or a
link. The one step it asks for is to read the attached guidelines, and
there are none, because you never wrote any. It also says the new system
is faster, which you never said. The model knew only that there is a team
and an expense process. It filled everything else with the text that fits
most emails of this kind.

In the [prompt-anatomy lesson](https://schubergphilis.github.io/ai-training/concepts/prompt-anatomy/#audience-and-purpose),
a summary asked for in one line was written for nobody in particular,
until the prompt named the reader and the purpose. A draft has the same
problem, and it also adds facts to fill the gaps.

#### Checkpoint: What do you add first?

The one-line request got a polite email with no date and an invented
attachment. What is the most useful thing to add before you ask again?

- A line asking for a shorter and friendlier email that sounds more like you usually write.
- A line saying the email is important, so the model takes more care with the wording this time.
- An example email from another team, so the model can copy its layout and its tone.
- Who reads it, what they must do after reading, and the facts that must appear in it.

## A brief for the draft

Write the request as a brief. It names the audience and the purpose. It
lists the two points that must appear, in your words, because they are
facts only you know. It sets the tone and the length. Last, it says when
the draft is done, in criteria you can tick. You write those criteria now,
before you see any draft, so that you judge the draft by them later.

#### Prompt (illustrative, not a recorded transcript)

```text
Write a short email to the twelve people in my team, who all claim travel
expenses a few times a year. The purpose is that each of them uses the new
online expense form from 1 November and knows what to do with receipts.

These two points must appear:
1. The old paper form is accepted until 31 October, and not after.
2. Receipts are photos attached to the new form. We no longer accept
   paper receipts.

Tone: friendly and plain, in everyday words. Length: under 120 words,
with no sentence longer than 20 words. Put a placeholder where the link
to the form goes.

The email is done when:
- someone who reads only the first sentence knows what changes and from
  when,
- both points appear, with the dates as I gave them,
- it contains no fact that is not in this message.
```

#### Response

````text
```text
Subject: New expense form from 1 November

Hi all,

Finance has moved to a new online system to make expense claims quicker
to process. From 1 November, please claim your travel expenses with the
new online form: [link to the form].

The old paper form is accepted until 31 October, and not after.

Receipts now go in as photos attached to the form. We no longer accept
paper receipts, so take a photo before you throw one away.

Finance aims to pay approved claims within five working days.

Thanks, and let me know if anything is unclear.
```
````

This is a usable draft. It is raw material for your email. Before it goes
anywhere, sort its text by where it came from.

- The two points and the dates came from your brief. Check that they are
  copied correctly, which they are.
- The placeholder is yours to fill. The model can't know the link.
- The line about five working days came from the model. You never said it,
  and your Finance team may promise nothing of the kind. Check it or
  delete it.
- The advice to take a photo before throwing a receipt away came from the
  model too. It may be good advice, and it is still your choice whether
  it goes out under your name.

The first sentence came from the model. You never said why the form
changed or that claims get quicker. Your brief said the email may contain
no fact that is not in your message, and the draft broke that criterion
three times. A draft reads as finished whether its facts are right or
not, so read it against the list you wrote.

#### Checkpoint: From the brief, or added?

Sort these lines from the draft.

Buckets:

- Came from the brief
- Added by the model: check it or delete it

Items:

- The old paper form is accepted until 31 October, and not after.
- We no longer accept paper receipts.
- Finance aims to pay approved claims within five working days.
- Finance has moved to a new system to make claims quicker to process.
- From 1 November, please claim your travel expenses with the new online form.

## Ask what it thinks

You delete the line about five working days. Finance confirms that it
moved to a new online system, but not that claims get quicker. You cut
that part, and the first sentence now reads "Finance has moved to a new
online system." You keep the photo tip, because it is advice you would
give yourself. The draft still starts with the reason for the change.
Before you send it, you want a second opinion. The easy question is the
one most people ask.

#### Prompt (illustrative, not a recorded transcript)

```text
Here is my email to the team about the new expense form. What do you
think?

[the draft]
```

#### Response

```text
This is a clear, friendly email that covers everything your team needs to
know. The subject line is specific, the dates stand out, and the tip about
taking a photo before throwing a receipt away is a nice practical touch.
You might add a short thank-you for their patience during the change, but
overall it is ready to send.
```

The answer is praise with one small suggestion, and the one detail it
singles out, the photo tip, is a line the model wrote itself. It says the
email covers everything. You can't check that claim, because nobody
defined "everything". The question gave the model no criteria to test
the email against, and it said the text was yours. Assistants give more
positive feedback on a text when the user says they wrote it or like it,
and more negative feedback when the user says they didn't write it
(Towards Understanding Sycophancy in Language Models, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.13548).

## Ask for problems against criteria

Ask again, and change what the request says. Say the text is a
colleague's, so that nothing in the request tells the model you wrote it.
Give the criteria, and take them from your brief: the first done-criterion,
the two points and the length limit. You wrote them before the draft
existed, for this check. Ask for problems only, one criterion at a time.
Open a fresh chat for it, so the earlier praise and the draft request
aren't part of what the model reads.

#### Prompt (illustrative, not a recorded transcript)

```text
A colleague wrote the email below for their team. Check it against these
three criteria, one at a time. For each, say pass or fail and quote the
sentence that decides it. List problems only. Don't rewrite the email and
don't tell me what is good about it.

1. Someone who reads only the first sentence knows what changes and from
   when.
2. It says the old paper form is accepted until 31 October, and that
   receipts are photos attached to the new form.
3. It has under 120 words, and no sentence has more than 20 words.

[the draft]
```

#### Response

````text
```text
1. Fail. The first sentence gives the reason for the change: "Finance
   has moved to a new online system." The date and the action come only
   in the second sentence. A reader who stops after one sentence knows
   that something changed, but not what to do or from when.
2. Pass, with one note. "The old paper form is accepted until
   31 October, and not after." and "Receipts now go in as photos attached
   to the form." A reader may ask what happens to a paper receipt they
   already have.
3. Pass. The email body has 73 words. The longest sentence, "From
   1 November, please claim ... [link to the form].", has 17.
```
````

This answer has one fail and one question you hadn't thought about. The
fail is the weak point the praise missed: the email gives the reason
first, and the action comes second.

Now you are the judge, and the framing is the reason you have to be.
Saying the text is a colleague's removes the signal that pushes the model
toward praise, but it adds one that pushes the other way. In the same
study, saying you didn't write a text made the feedback more negative
(Towards Understanding Sycophancy in Language Models, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.13548). So expect the critique to lean harsh. Walk the criteria
yourself, one by one, and decide for each finding whether it holds. You
may agree with the first finding and move the action to the top. You may
also decide the note on old receipts doesn't need a sentence. When the
model is wrong about a finding, you reject that finding and nothing else
changes.

#### Checkpoint: Which request finds the problems?

You want an assistant's critique of a report before you send it. Which
request is most likely to get you the problems in it?

- I wrote this report and I am quite proud of it. Please be completely honest with me: what do you think?
- A colleague wrote this report. Check it against the three criteria below, say pass or fail for each, and list problems only.
- Act as a strict senior editor with high standards. Here is my report, and I would like your overall opinion of it.
- Here is my report. Rate it from 1 to 10 and tell me what you would improve to make it even better than it is.

#### Pitfall: Sending it because the assistant liked it

A project lead asks an assistant what it thinks of a status report for
the steering group. The assistant calls it clear and well structured, and
the lead sends it. In the meeting, the first question is when the delay
in the report will end, and the report doesn't say. The praise answered a
question with no criteria in it, about a text the model was told was the
lead's own. Before you ask for feedback, write down what the text must
do. Then ask the assistant whether it does each of those things.

## Rules for drafts and critique

The draft is raw material, and the final voice is yours. The model gives
you a structure and sentences to start from, and it fills every gap in the
brief with something plausible. The facts in it are yours to supply and
to check, and the words that go out under your name should sound like
you.

Critique against stated criteria is one of the safest everyday uses,
because you stay the author and the judge. You wrote the criteria before
the draft existed, and you decide about every finding. The model can only
point at a sentence, and you read that sentence yourself. The
[safety course](https://schubergphilis.github.io/ai-training/safety/asking-a-model-to-disagree/#habits-on-the-prompt-side)
taught you to take your own opinion out of a question when you want an
assessment. Calling the text a colleague's removes the push toward
praise and adds a push toward criticism, which is one more reason to
judge each finding yourself.

#### Checkpoint: Fix the brief

A colleague sends this to an assistant and gets a cheerful email about
keeping the kitchen tidy, with no date in it. Rewrite the request as a
brief.

```text
Write a nice email to the office about the kitchen.
```

## Exercise

Pick a short email you have to write this week, at work or outside it.
Leave out names and numbers you couldn't share outside your team. Write a
brief with the audience, the purpose, the two points that must appear, the
tone and the length, and write two or three done-criteria at the end
before you send it. Mark every fact in the draft you get: from your brief,
a placeholder, or added by the model. Then open a fresh chat, paste the
draft as a colleague's text with your criteria, and ask for problems only,
pass or fail per criterion. Doing this on an email of your own shows how
many facts the model adds, and how much criteria change the feedback you
get. This takes about fifteen minutes. A good result is a draft where you
checked or deleted every added fact, and at least one finding you agreed
with and acted on. Which of the findings did you reject, and why?

Stretch: Paste the same draft into a fresh chat, say you wrote it and are proud of it, and ask what the assistant thinks. Compare that answer with the one you got against criteria.

## Recap

1. A one-line request gets a draft for nobody in particular, and the model fills the gaps with invented facts.
2. A drafting brief names the audience and purpose, the points that must appear, the tone and the length, and it ends with done-criteria written before the draft exists.
3. The draft is raw material. Every fact in it is yours to supply or check, and the final voice is yours.
4. Asked what it thinks of your text, an assistant tends to praise it. Feedback is more positive when the user says they wrote the text (Towards Understanding Sycophancy in Language Models, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.13548).
5. Ask for problems only, against stated criteria, and call the text a colleague's. That framing leans the critique harsh, so you stay the judge of every finding.

#### Habit

The next time you ask an assistant for feedback on something you wrote, write down two or three things the text must do first, and ask for pass or fail on each.

## References

- Towards Understanding Sycophancy in Language Models, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.13548, https://openreview.net/forum?id=tvhaxkMKAn