# Reasoning before each action

> Give the fake model a one-line thought before each tool request, compare its trace with the plain loop's on the same tasks, and see when the reasoning changes the call and what it costs.

Page: https://schubergphilis.github.io/ai-training/building-agents/react/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson the fake model writes one line of reasoning before each
tool request, and the loop prints that line in the trace. We run one task
with the model that writes the line and with one that doesn't, and compare
the traces. After that we look at a task where the line changes which tool
the model asks for, and at what the line costs.

The loop is the one from
[Stopping the loop on purpose](https://schubergphilis.github.io/ai-training/building-agents/stopping-the-loop/),
with the same `run` and the same stop rules. Its `show` prints one more
line, the thought, when a request has one. The complete
program is `site/examples/building-agents/react/agent.py` in the
repository. The tools belong to a small shop. `get_order` returns the
item and the days since delivery, `get_policy` returns the text of a shop
policy, and `issue_refund` refunds an order in full.

## One line of reasoning before each call

The pattern has a name, ReAct, from the paper that tested it: the model
alternates a short written reasoning step with an action, reads the
result, and reasons again (ReAct: Synergizing Reasoning and Acting in Language Models, International Conference on Learning Representations (ICLR 2023), arXiv preprint 2210.03629). In the paper, the reasoning helps
the model keep and update a plan, and handle surprises, and the actions
fetch the facts that the reasoning needs (ReAct: Synergizing Reasoning and Acting in Language Models, International Conference on Learning Representations (ICLR 2023), arXiv preprint 2210.03629). The lesson on
agentic design patterns in the Agent Engineer Course presents it as a
loop of think, act, observe (Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course).

In the fixture, the reasoning is one more field in the reply. The fake
model `model_reasons` writes a `thought`, then asks for the tool that
follows from it. Here is its first round:

```python
if len(done) == 0:
    if question.startswith("Refund"):
        thought = "A refund is allowed only inside the return window, so I check the order first."
    else:
        thought = "A return depends on when the order arrived, so I look up the order."
    return {"thought": thought, "tool": "get_order", "args": {"order_id": order_id}}
```

In the second round it reads the days since delivery from the result and
asks for the return policy. After that it compares the two numbers and
answers. `show` prints `thought:` and the text before the round line of
every request that has one.

#### Example: The trace with thoughts

Run this, and compare what you see with the output below.

```python
show(run("Can I still return order 1042?", model=model_reasons))
```

Output:

```text
user: 'Can I still return order 1042?'
thought: A return depends on when the order arrived, so I look up the order.
round 1: get_order(order_id='1042') -> {'ok': True, 'item': 'kettle', 'days_since_delivery': 12}
thought: It arrived 12 days ago. I need the return window to compare.
round 2: get_policy(topic='returns') -> {'ok': True, 'text': 'Returns and refunds within 30 days of delivery.'}
stop: end_turn
answer: Yes. The kettle arrived 12 days ago, inside the 30-day window.
```

Two rounds, each with the reason written before it. The final reply is
an answer with no tool request, so `run` returns before a third round and
the answer has no thought line in front of it.

## The same run without the reasoning

`model_acts` is the kind of model the loop had before this lesson. It
replies with a tool request only, and it picks the tool from the words of
the question. Run it on the same task.

#### Example: The plain loop on the same task

Run this, and compare what you see with the output below.

```python
show(run("Can I still return order 1042?", model=model_acts))
```

Output:

```text
user: 'Can I still return order 1042?'
round 1: get_order(order_id='1042') -> {'ok': True, 'item': 'kettle', 'days_since_delivery': 12}
round 2: get_policy(topic='returns') -> {'ok': True, 'text': 'Returns and refunds within 30 days of delivery.'}
stop: end_turn
answer: Yes. The kettle arrived 12 days ago, inside the 30-day window.
```

The calls, their order, their results and the answer are all the same.
The trace with thoughts adds one thing, a reason you can read for each
call. On this task, a person who debugs the run learns why the
agent looked up the order, and the agent does nothing different.

## When the reasoning changes the call

The second task asks for a refund of order 1077, a lamp delivered 45 days
ago. `model_acts` sees the word "Refund" and asks for the refund tool.

#### Example: A refund without reasoning

Run this, and compare what you see with the output below.

```python
show(run("Refund order 1077.", model=model_acts))
```

Output:

```text
user: 'Refund order 1077.'
round 1: issue_refund(order_id='1077') -> {'ok': True, 'refunded': '1077'}
stop: end_turn
answer: Done. Order 1077 is refunded.
```

The refund went through for an order outside the 30-day window, and the
model didn't read the policy. `model_reasons` writes
its first thought about what a refund depends on, and that thought leads
to a different first call.

#### Example: A refund with reasoning

Run this, and compare what you see with the output below.

```python
show(run("Refund order 1077.", model=model_reasons))
```

Output:

```text
user: 'Refund order 1077.'
thought: A refund is allowed only inside the return window, so I check the order first.
round 1: get_order(order_id='1077') -> {'ok': True, 'item': 'lamp', 'days_since_delivery': 45}
thought: It arrived 45 days ago. I need the return window to compare.
round 2: get_policy(topic='returns') -> {'ok': True, 'text': 'Returns and refunds within 30 days of delivery.'}
stop: end_turn
answer: The lamp arrived 45 days ago, outside the 30-day window. I did not refund it.
```

Both fake models are written by hand to show the difference. The
fixture proves nothing about real models. The paper measured it. Adding
the reasoning steps did better than the same prompts with actions only,
on question answering and fact checking, and on the two interactive
tasks ReAct beat methods trained by imitation and reinforcement learning
(ReAct: Synergizing Reasoning and Acting in Language Models, International Conference on Learning Representations (ICLR 2023), arXiv preprint 2210.03629).

The line has a price. The thoughts stay in the message list, and the
loop sends the whole list to the model again in every round. The `cost`
step runs the return task with each model and prints the size of the
message list after the last round.

#### Example: The size of the message list

Run this, and compare what you see with the output below.

```python
for name, model in (("model_acts", model_acts), ("model_reasons", model_reasons)):
    messages = run("Can I still return order 1042?", model=model)["messages"]
    print(f"{name}: {len(str(messages))} characters")
```

Output:

```text
model_acts: 466 characters
model_reasons: 623 characters
```

The calls and the answer are the same, and the list is longer by the two
thoughts. A real model writes each line as output tokens, and the loop
sends the line back as input in every later round of the run.

## The loops in this course already alternate

The loops in this course already alternate a model decision with an
action and its result: the model decides on an action, the loop runs it,
and the model decides the next action from the result. The pattern adds
the written reason between the result and the next decision. Some models
write that reason on their own. On Claude, a model that supports
interleaved thinking can write its reasoning after each tool result and
before the next call. The API returns that reasoning as a summary, or as
an empty field when the `display` setting is `omitted`, and it bills the
full reasoning in both cases (Thinking, Claude Platform documentation). The choice you
make as the builder is whether the reasoning appears in a form you can
read and keep.

A task gains from the extra tokens when its next step depends on what an
earlier result said. The agent then chooses between tools as it goes.
The refund is such a task: the order decides whether the refund tool is
right. When the calls are the same whatever the results say, as with one
export per table every night, the line doesn't change a call, and a
fixed script or a plan fits the job better. People who review or debug
the runs have a second reason to keep the line, because it shows them
where the agent went wrong.

#### Checkpoint: Which task gains from the reasoning line?

Which of these tasks gains the most from one line of reasoning before
each tool call?

- Rename 300 photos by the same date rule, with one tool call per photo.
- Read a customer's plan and payment history, then choose a refund, a credit or a handover to a person.
- Translate a product description into French, with no tools at all.
- Report the stock level of one item from one tool that is always the only call.

#### Pitfall: Reading the thought as a record of what happened

A reviewer sees "I check the order first" in a trace and concludes that
the agent checked the order. The thought is text the model wrote. In the
fixture's `run`, the loop runs `reply["tool"]` with `reply["args"]` and
never reads `reply["thought"]`, so a thought and a call can disagree and
the loop runs the call. Research on written reasoning found that models
leave out factors that changed their answer and write plausible reasons
for answers they were pushed toward (Language Models Don't Always Say What They Think, Advances in Neural Information Processing Systems 36 (NeurIPS 2023)). On Claude, the
thinking you get back is a summary or an empty field, and no display
setting returns the raw reasoning (Thinking, Claude Platform documentation). Rule: check
what the agent did in the round lines, the calls and their results, and
read the thought as the model's explanation of them.

## Exercise

Copy `site/examples/building-agents/stopping-the-loop/agent.py` from the
repository. Add a `thought` field to the replies of its three fake
models, `model_never_answers`, `model_retries_forever` and
`model_alternates`, and change its `show` to print the thought before
each round line, the way this lesson's `show` does. Run the
`never_answers`, `retries` and `alternates` steps, and put each trace
next to the one the original file prints. Doing it once on a loop you
already know shows how little the pattern changes in the code.

A good result: three pairs of traces with the same rounds and the same
stop reasons, where the new trace of each pair has a thought line before
every round. Give `model_retries_forever` the same thought in every round
for that. Then put the round number in its thought and run `retries`
again. Which rule ends the run now, and what in `repeated` did the
thought change?

Stretch: Give model_alternates a thought that names the fix it tried last and the result it saw, and check whether a reader of that trace could spot the alternation before the step limit.

## Recap

1. ReAct alternates a short reasoning step with an action and reads the
   result before it reasons again (ReAct: Synergizing Reasoning and Acting in Language Models, International Conference on Learning Representations (ICLR 2023), arXiv preprint 2210.03629).
2. On a task with a fixed path, the reasoning line adds a readable reason
   for each call and leaves the calls the same.
3. When the next tool depends on an earlier result, the reasoning step can
   change which tool the model asks for, at the cost of one more line per
   step that the loop sends back in every round.
4. A tool-using loop already alternates decisions and actions, and on
   Claude a thinking model can reason between tool calls whether or not
   the loop shows it (Thinking, Claude Platform documentation). A shown thought can leave
   out what drove the choice (Language Models Don't Always Say What They Think, Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), and the round lines are the
   record of what the loop ran.

## References

- ReAct: Synergizing Reasoning and Acting in Language Models, International Conference on Learning Representations (ICLR 2023), arXiv preprint 2210.03629, https://openreview.net/forum?id=WE_vluYUL-X
- Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/04-agentic-design-patterns/
- Thinking, Claude Platform documentation, https://platform.claude.com/docs/en/build-with-claude/thinking
- Language Models Don't Always Say What They Think, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), https://proceedings.neurips.cc/paper_files/paper/2023/hash/ed3fea9033a80fea1376299fa7863f4a-Abstract-Conference.html