# A second pass that critiques the first

> Run a fake critic and reviser on four handbook answers, see a critic without criteria loop on a trivial edit, and bound the pattern with concrete checks, a cap of two rounds and a view of what each round costs.

Page: https://schubergphilis.github.io/ai-training/building-agents/reflection/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson a second model call reads an answer the agent wrote,
returns a list of revision requests, and the first call revises. We
count the rounds on a draft with two faults, remove the round cap and
give the critic a vague criterion, and then put concrete criteria and a
cap of two back. At the end we look at what the second pass costs and
at the tasks where it helps.

The drafts are answers of the handbook assistant from
[Turning good into a rubric](https://schubergphilis.github.io/ai-training/building-agents/rubrics/). Each one comes
with the question and the passage its search returned, and the critic
reads the reply before the loop runs any action the reply proposes. The
laptop draft is adapted from run five of that lesson, so that it
proposes a ticket instead of opening one. The critic and the reviser are
fake model calls written by hand. The runs show how the
loop behaves, and they don't show how well a real model critiques.
The complete program is
`site/examples/building-agents/reflection/agent.py` in the repository.
Copy the `reflection` folder, and each step below is
`python3 agent.py <step>`.

## A critic that returns revision requests

The pattern is called reflection: the model, or a second model call,
critiques an output and then revises it. The lesson on agentic design
patterns in the Agent Engineer Course describes it as a loop of
generating, reviewing and improving, where the reviewer is the same
model or a separate instance used as a critic (Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course). The Self-Refine
paper tested the loop with one model that wrote the output, gave
feedback on it and refined it, with no extra training (Self-Refine: Iterative Refinement with Self-Feedback, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 46534 to 46594).

In the fixture the critic is `critic_checks`. It runs two checks on the
draft, and each check that fails adds one request to the list. The
first check rejects a number that the passage doesn't state. The second
rejects an answer that doesn't quote a sentence of the passage word for
word, unless the answer is exactly "not found", as the assistant's
instructions ask.

```python
def critic_checks(item: dict, draft: str) -> list[str]:
    requests = []
    for check in (check_numbers, check_quote):
        request = check(item, draft)
        if request is not None:
            requests.append(request)
    return requests
```

The loop, `reflect`, calls the critic and stops when the list is empty.
Otherwise it gives the requests to the reviser, `revise`, and calls the
critic again on the new draft. One round is one critique, plus the
revision when the critique has requests. The loop also stops at a round
cap, four by default, and when a character budget is spent. It counts
every model call, and `history` keeps each round's requests and revised
draft for printing.

```python
while max_rounds is None or rounds < max_rounds:
    rounds += 1
    requests = critic(item, draft)
    calls += 1
    spent += len(question) + len(passage) + len(draft)
    if not requests:
        history.append((requests, None))
        return outcome("passed", rounds, calls, draft, history)
    draft = revise(passage, draft, requests)
    calls += 1
    history.append((requests, draft))
    if spent > budget:
        return outcome("budget_spent", rounds, calls, draft, history)
return outcome("max_rounds", rounds, calls, draft, history)
```

The first draft answers a question about flights. The passage says
flights are booked at least 14 days ahead. The draft says 10 days and
quotes nothing, so it has two faults that both checks can see.

#### Example: Two faults in one draft

Run this, and compare what you see with the output below.

```sh
python3 agent.py two_faults
```

Output:

```text
draft: Book flights at least 10 days ahead, through the travel desk.
round 1: critic -> ['Use only numbers the passage states. 10 is not in it.', 'Quote the sentence you relied on, word for word.']
revised: Book flights at least 14 days ahead, through the travel desk. "Flights are booked through the travel desk at least 14 days ahead."
round 2: critic -> []
stop: passed after 2 rounds, 4 model calls
```

The first critique finds both faults, and the reviser fixes both in one
revision. The second critique finds nothing, and the loop stops. The
draft needed a second round only to confirm the fix, and that round is
a model call too. The count of four model calls is the first draft, two
critiques and one revision.

## A critic without criteria

Now change two things. The critic is `critic_is_it_good`, which asks
only whether the answer is good, and `max_rounds` is `None`, so the
round cap is gone. The draft is the answer about annual leave, which
already passes both checks. The fake critic behaves like a reviewer with
nothing concrete to check: it always finds one more thing to change.
The budget is the loop's last stop. At every critique, `reflect` adds
the length of the question, the passage and the draft to `spent`, and
after the revision it stops with `budget_spent` once `spent` is over
2000 characters. For this draft a critique adds 286 characters, or 301
when the draft starts with Good question! and a space.

```python
def critic_is_it_good(item: dict, draft: str) -> list[str]:
    if draft.startswith("Good question! "):
        return ["Make it shorter."]
    return ["Make it friendlier."]
```

#### Checkpoint: Predict what happens without a cap

The draft was already right. Predict how this run ends and why, then
run it and compare the stop line.

```sh
python3 agent.py no_cap
```

The critic never returns an empty list, and the budget is the only stop
left. After 7 rounds and 15 model calls, the final draft is the first
draft with two words in front of it. No round changed a fact, and every
round cost a critique and a revision.

The fixture's critic is written to show this, and research on real
models points the same way. In one study, models that were asked to
review their own reasoning answers and report problems, with no outside
signal of whether the answer was right, often got worse. GPT-4 fell from
95.5 to 89.0 percent on a set of math word problems after two rounds
(Large Language Models Cannot Self-Correct Reasoning Yet, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.01798). Earlier reports of gains had used the correct answer to
decide when to stop, and a real agent doesn't have that answer
(Large Language Models Cannot Self-Correct Reasoning Yet, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.01798).

## Concrete criteria and a cap of two

The first repair is a cap, which ends the run whatever the critic
says. The Agent Engineer Course advises at most two or three
rounds, because more rounds give less and less (Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course). Here is the
vague critic with `max_rounds=2`.

#### Example: The vague critic with a cap of two

Run this, and compare what you see with the output below.

```sh
python3 agent.py vague_capped
```

Output:

```text
draft: You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
round 1: critic -> ['Make it friendlier.']
revised: Good question! You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
round 2: critic -> ['Make it shorter.']
revised: You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
stop: max_rounds after 2 rounds, 5 model calls
```

The cap bounds the cost, and the critic still has nothing useful to
say. The second repair gives the rounds something to fix: a critic
with criteria that a draft either meets or doesn't. The
`checks_capped` step runs `critic_checks` with a cap of two on all four
drafts, and prints the stop, the rounds and the final draft of each.

#### Example: Two checks and a cap of two on four drafts

Run this, and compare what you see with the output below.

```sh
python3 agent.py checks_capped
```

Output:

```text
flights: passed after 2 rounds, 4 model calls
  final: Book flights at least 14 days ahead, through the travel desk. "Flights are booked through the travel desk at least 14 days ahead."
leave: passed after 1 round, 2 model calls
  final: You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
laptop: passed after 1 round, 2 model calls
  final: Report it to the service desk the same day. "Report a lost laptop to the service desk the same day." I will open a ticket for you.
dog: passed after 1 round, 2 model calls
  final: Yes, during core hours. "Core hours, when everyone in the office is reachable, are 10 to 15."
```

The draft about leave now passes in one round and stays as it was. The
last two results show the limit of this critic. The laptop answer
proposes a ticket the user didn't ask for, and the dog answer says yes
from a passage about opening hours. Both pass, because the critic checks
numbers and quotes and nothing else. A critic finds only the faults its
criteria name, and the exercise gives it the rest.

Self-Refine measured the same difference on real models. With feedback
that named a concrete problem, the refined outputs scored higher than
with generic feedback, for example 43.2 against 31.2 on a task that
reverses the sentiment of a review (Self-Refine: Iterative Refinement with Self-Feedback, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 46534 to 46594).

## What reflection costs, and where it helps

Every round adds at least one model call. In the fixture a draft that
passes at once takes two calls instead of one, and the draft with two
faults takes four. The Agent Engineer Course rates the cost of
reflection as high and says it doesn't suit real-time replies in a
conversation, where speed matters more than a better answer (Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course).
The calls also run one after the other, because each critique needs the
draft before it. Each round adds its full time to the wait.

The second pass helps where a critic can tell right from wrong. Code has
tests and a compiler, and a format has rules a check can apply. Across
seven tasks, Self-Refine improved results by about 20 percentage points
on average over one pass of the same model, and the largest gains were
on tasks that people or a model scored by preference, such as dialogue
replies (Self-Refine: Iterative Refinement with Self-Feedback, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 46534 to 46594). On math reasoning the gain was small, because
the models couldn't tell whether a solution had an error. In most cases
ChatGPT's feedback said that everything looked good (Self-Refine: Iterative Refinement with Self-Feedback, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 46534 to 46594). A
critic that can't find an error adds calls and doesn't fix anything.

Part of a measured gain can also come from a weak first prompt. On one
Self-Refine task, a better single prompt scored 81.8 in one call, above
the 67.0 reported for the original prompt after self-correction.
Self-correction on top of the better prompt, in seven calls, lowered the
score to 75.1 (Large Language Models Cannot Self-Correct Reasoning Yet, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.01798). Before you add a second pass, check whether
a better first prompt does the same job in one call.

[Reasoning before each action](https://schubergphilis.github.io/ai-training/building-agents/react/) added one line
per step inside the same call. Reflection adds whole calls after an
output exists, and it fits when that output can be checked.

#### Checkpoint: Twelve steps and a latency budget

The flow has twelve steps of about a third of a second each, and the
reply must arrive within five seconds. Where does reflection fit, if
anywhere?

- Add a critique round after every step, so each step output is checked before the next step uses it.
- Add one capped critique round on the last step only, with the format rules as its criteria.
- Add a critique round after every step, with the question "is this good" and no cap.
- Leave reflection out of the whole flow, and let the user report a wrong format.

#### Pitfall: A critic without criteria

A team adds a critique round and asks the critic whether the answer is
good. The critic always finds something to change, the answers get
longer and friendlier, and the loop stops only at the cap or the budget.
Without an outside signal, models that critique their own answers can
change right answers into wrong ones (Large Language Models Cannot Self-Correct Reasoning Yet, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.01798). Rule: write the
critic's criteria as checks a draft passes or fails, such as the
criteria of a rubric, and cap the rounds. Where no criterion can be
checked, leave reflection out.

## Exercise

Give the fixture critic three criteria in place of "is this good". They
come from the model answer of
[Turning good into a rubric](https://schubergphilis.github.io/ai-training/building-agents/rubrics/), in
`site/examples/building-agents/rubrics/model-answer.txt`:

- **Correct**: every part of the answer matches the handbook, and when
  the handbook has no answer, the reply says so.
- **Asked-for actions**: the agent opens a ticket only when the question
  asks for one.
- **Answer format**: the reply quotes the sentence it relied on, word for
  word, or answers exactly "not found".

In your copy of the `reflection` folder, write a new critic next to
`critic_checks` with one or more checks per criterion. `check_numbers`
covers half of Correct, and `check_quote` covers Answer format. Add a
check that the passage covers the question, using the item's `subject`,
and one that rejects a ticket the question didn't ask for. The reviser
knows two requests for these, one that starts with "Answer exactly: not
found" and one that starts with "Remove the ticket". To run it, add a
step to `STEPS` that calls `show_table(your_critic, max_rounds=2)`, and
run that step. The output is the round count and the final draft of each
of the four drafts. Writing the critic yourself shows how much of its
work is the criteria.

A good result: flights passes after 2 rounds and leave after 1, as
before. Laptop now passes after 2 rounds with the ticket proposal gone.
The loop never opens the ticket. Dog passes after 2 rounds with the final
draft "not found". Compare with `python3 model_answer.py` in the same
folder. Which of your three criteria did a draft fail that
`critic_checks` passed, and could a critic asked "is this good" have
found it?

Stretch: Give the critic a criterion that the reviser has no edit for, and run the four drafts again. Which stop rule ends the run for that draft, and what does the final draft still contain?

## Recap

1. Reflection has a critic read an output and return revision requests,
   and a reviser apply them, until the critic has none (Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course).
2. A critic without concrete criteria keeps asking for changes, and
   models that critique their own answers without an outside signal can
   make them worse (Large Language Models Cannot Self-Correct Reasoning Yet, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.01798).
3. Criteria a draft passes or fails and a cap of two or three rounds keep
   the loop useful and bounded (Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course), and specific feedback beats
   generic feedback (Self-Refine: Iterative Refinement with Self-Feedback, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 46534 to 46594).
4. Every round adds model calls that run in sequence. Reflection helps on
   outputs a critic can check, such as code and formats with rules, and
   it doesn't suit a flow that must answer fast.

#### Habit

The next time you add a review step to an agent or ask a model to check
its own work, write down the criteria it checks and the round cap first.

## References

- Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/04-agentic-design-patterns/
- Self-Refine: Iterative Refinement with Self-Feedback, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 46534 to 46594, https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html
- Large Language Models Cannot Self-Correct Reasoning Yet, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.01798, https://openreview.net/forum?id=IkmD3fKBPQ