# Writing the plan before taking the steps

> Have the fake model write a numbered plan before any tool runs, work through it, and see a stale plan send a false email until the model revisits the plan after each step.

Page: https://schubergphilis.github.io/ai-training/building-agents/planning/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson the fake model writes a numbered plan for a goal before
any tool runs. We print the plan and tick off each step as it runs.
Then step three returns a fact the plan didn't expect,
and we compare a run that follows the old plan with a run where the
model revisits the plan after each step. At the end we look at what the
written plan gives a person who reads it before the first step, and at
what it costs.

The tools belong to the small shop from
[Reasoning before each action](https://schubergphilis.github.io/ai-training/building-agents/react/), with three new
ones. `check_stock` returns how many of an item the shop has,
`send_replacement` books a replacement for an order, and
`email_customer` sends the customer a message. The fake model and the
code that runs the plan are written by hand. The runs show how the
pattern behaves, and they don't show how well a real model plans. The complete
program is `site/examples/building-agents/planning/agent.py` in the
repository. Copy the `planning` folder, and each step below is
`python3 agent.py <step>`.

## A plan before the first step

The pattern is called planning: the agent writes out the steps to a goal
before it takes them, as a list it then works through and updates. The
lesson on planning in the Agent Engineer Course compares it to a packing
list that you write before a trip and tick off as you pack (Planning and reasoning: the loop, plan-then-execute, hierarchy, Agent Engineer Course).

The goal is "Replace the broken kettle from order 1042." The fake
model's first reply is the whole plan. Each step has a line of text that
a person can read and the tool call that does the step.

```python
return [
    make_step(f"Look up order {order_id}", "get_order", order_id=order_id),
    make_step("Read the returns policy", "get_policy", topic="returns"),
    make_step(f"Check that a {item} is in stock", "check_stock", item=item),
    make_step(f"Send a replacement {item}", "send_replacement", order_id=order_id),
    make_step(
        f"Email the customer that a new {item} has shipped",
        "email_customer",
        order_id=order_id,
        message=f"Your new {item} has shipped.",
    ),
]
```

`show_plan` prints the list with a box in front of each step, and an `x`
in the box of each step that has run.

#### Example: The plan for the kettle

Run this, and compare what you see with the output below.

```sh
python3 agent.py plan
```

Output:

```text
goal: Replace the broken kettle from order 1042.
plan v1:
  1. [ ] Look up order 1042
  2. [ ] Read the returns policy
  3. [ ] Check that a kettle is in stock
  4. [ ] Send a replacement kettle
  5. [ ] Email the customer that a new kettle has shipped
```

No tool has run yet. The plan is text in the fake model's reply, and
the shop is as it was.

## Working through the list

The plan runs one step at a time. For each step the code calls the tool
the step names, keeps the result and prints a step line. After a step
has run, its box gets an `x`. The `progress` step runs the first two
steps and prints the plan at that point.

#### Example: The plan after two steps

Run this, and compare what you see with the output below.

```sh
python3 agent.py progress
```

Output:

```text
step 1: get_order(order_id='1042') -> {'ok': True, 'item': 'kettle', 'days_since_delivery': 12}
step 2: get_policy(topic='returns') -> {'ok': True, 'text': 'Returns and refunds within 30 days of delivery.'}
plan v1:
  1. [x] Look up order 1042
  2. [x] Read the returns policy
  3. [ ] Check that a kettle is in stock
  4. [ ] Send a replacement kettle
  5. [ ] Email the customer that a new kettle has shipped
```

The printed plan shows where the run is and what comes next, in words.
A loop without a plan shows only the calls it has made so far. The
Agent Engineer Course lists this progress view among the reasons to plan,
next to steps that a plan keeps from being skipped (Planning and reasoning: the loop, plan-then-execute, hierarchy, Agent Engineer Course).

Research on prompts measured the second reason. The Plan-and-Solve paper
replaced the prompt "Let's think step by step" with one that asked the
model to understand the problem, write a plan and then carry it out.
The authors wanted fewer answers that skip a reasoning step. With GPT-3
the plan-first prompt did better on every set of math word problems they
tested, and its average accuracy was 2.5 percentage points higher, 72.9 against
70.4 (Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2023), 2609-2634). In
a sample of 100 problems, the wrong answers that skipped a step went
from 12 with the old prompt to 10 with the plan-first prompt, and to 7
with a longer version of it (Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2023), 2609-2634). That plan was written and
followed inside one reply, with no tools, and the model didn't revise it
after it started.

## A step that changes the facts

`follow` runs every step of the first plan in order. It makes one model
call for the plan. After that it runs the tools in order. Here is
the whole run.

#### Example: Following the first plan

Run this, and compare what you see with the output below.

```sh
python3 agent.py follow
```

Output:

```text
step 1: get_order(order_id='1042') -> {'ok': True, 'item': 'kettle', 'days_since_delivery': 12}
step 2: get_policy(topic='returns') -> {'ok': True, 'text': 'Returns and refunds within 30 days of delivery.'}
step 3: check_stock(item='kettle') -> {'ok': True, 'item': 'kettle', 'in_stock': 0, 'restock_days': 21}
step 4: send_replacement(order_id='1042') -> {'ok': True, 'replacement_for': '1042', 'status': 'backordered'}
step 5: email_customer(order_id='1042', message='Your new kettle has shipped.') -> {'ok': True, 'sent_to': 'customer of 1042'}
plan v1:
  1. [x] Look up order 1042
  2. [x] Read the returns policy
  3. [x] Check that a kettle is in stock
  4. [x] Send a replacement kettle
  5. [x] Email the customer that a new kettle has shipped
model calls: 1
```

Every step reports `'ok': True`, and every box has an `x`. Step three
said that the shop has no kettles for 21 days. The plan was written
before anyone knew that, so step four booked a replacement that went on
back order, and step five told the customer a new kettle has shipped.
The plan was right when it was written and wrong after step three.

`work` runs the same plan in a different way. After each step it calls
the model again with the plan and every result so far, and the model
either keeps the plan or returns a new one. The steps that have run stay
as they are, and the model may change the steps still to come. Here is the
fake model's reply to that call.

```python
latest = results[-1]
if "in_stock" in latest and latest["in_stock"] == 0:
    order_id = plan[0]["args"]["order_id"]
    item, days = latest["item"], latest["restock_days"]
    return [
        *plan[: len(results)],
        make_step(f"Refund order {order_id}", "issue_refund", order_id=order_id),
        make_step(
            f"Email the customer that the {item} is refunded",
            "email_customer",
            order_id=order_id,
            message=f"The {item} is out of stock for {days} days, so we refunded you.",
        ),
    ]
return None
```

When the model returns a changed plan, `work` gives it the next
version number, so the first plan is v1 and the first revision is v2.
`show_plan` prints a `plan vN:` line and then one numbered line per
step, with `[x]` for a step that has run and `[ ]` for an open one. The
`revise_only` step runs the first three steps, gives the model the
results and prints only the plan it gets back.

#### Checkpoint: Predict the revised plan

What is the plan after the model revisits it? Predict the output, then
run it.

```sh
python3 agent.py revise_only
```

The first three steps keep their `x`, and the version number tells a
reader that the plan changed. Steps four and five of plan v2 run next,
and the customer gets a refund and a message that says why. The
`revise` step prints the plan before and after, with the result of step
three between them.

#### Example: The plan before and after

Run this, and compare what you see with the output below.

```sh
python3 agent.py revise
```

Output:

```text
before:
plan v1:
  1. [x] Look up order 1042
  2. [x] Read the returns policy
  3. [x] Check that a kettle is in stock
  4. [ ] Send a replacement kettle
  5. [ ] Email the customer that a new kettle has shipped
step 3 returned: {'ok': True, 'item': 'kettle', 'in_stock': 0, 'restock_days': 21}
after:
plan v2:
  1. [x] Look up order 1042
  2. [x] Read the returns policy
  3. [x] Check that a kettle is in stock
  4. [ ] Refund order 1042
  5. [ ] Email the customer that the kettle is refunded
```

The revised run has a cost. The `cost` step runs the kettle goal with
`follow` and with `work` and counts the calls.

#### Example: The calls of each run

Run this, and compare what you see with the output below.

```sh
python3 agent.py cost
```

Output:

```text
follow: 1 model call, 5 tool calls
work: 6 model calls, 5 tool calls
```

Both runs make five tool calls. `work` makes a model call for the plan
and one more after each step, and one of those five calls changed the
plan. The Agent Engineer Course suggests a light plan of a few steps,
run step by step, with a check of the plan after each major step
(Planning and reasoning: the loop, plan-then-execute, hierarchy, Agent Engineer Course). A cheaper `work` could call the model again only when a
result isn't what the plan expected, and the exercise asks for that as
a stretch.

## The moment to correct the plan

Before step one runs, the plan is five lines of text and the shop is
unchanged. A person who reads it then can fix it for the price of
reading it. Say the customer's message said "I want my money back". The
person reads step four, "Send a replacement kettle", and changes it
before any tool runs. After a run of `follow`, the same fix means a
cancelled back order and a second email that corrects the first.

Some agent products stop at this moment on purpose. Claude Code has a
plan mode for it. In that mode the agent may look around the project
first, and it may not change your files until you accept the plan it
shows you. The one exception is an interactive terminal session in
which bypass permissions are available. You can
also reply that it should keep planning and say what to change
(Choose a permission mode, Claude Code documentation). The written plan is what makes that
stop useful: a person can say no to a step while saying no costs
nothing.

#### Checkpoint: When is the fix cheapest?

The plan's first step picks the wrong audience. When can the person fix
it at the lowest cost?

- After the first batch of step two, from the progress of the plan.
- After the run, from the trace of tool calls and their results.
- At the next evaluation run, when the team reviews last month's sends.
- Before step one runs, by changing the audience in the written plan.

## When to write the plan first

Write the plan first for a task with steps that depend on each other,
where a wrong early step wastes the steps after it. A plan also helps
when a person or code should read the steps before they run. The kettle goal is
such a task: five steps, two of them change things for the customer,
and a person may want to approve them. The Agent Engineer Course advises
against a plan for a simple task. Its lesson on design patterns names
tasks of one step as a poor fit (Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course), and its lesson on
planning prefers a reactive approach for tasks of fewer than three steps
(Planning and reasoning: the loop, plan-then-execute, hierarchy, Agent Engineer Course). For those, the plan costs a call and says what the
first call would have done anyway.

A written list is also one choice among several for keeping track.
Claude Code has tools for a task list, and on newer models it leaves
them out by default. The documentation says a newer model keeps track
of work with many steps without the list, and that the tool definitions
take up space in the context (Tools reference, Claude Code documentation). The plan
in this lesson is written for someone to read it: the person who
approves it, and the code that runs it step by step.

#### Checkpoint: Which task gets a written plan first?

Which of these tasks gains the most from a plan that the agent writes
before it takes the first step?

- Look up the opening hours of one shop with one search tool and report them.
- Answer a greeting in a support chat, with no tools and no steps to take.
- Close a customer account: export the data, cancel two contracts, refund the balance and email a summary.
- Convert one date in a message from one format to another with a fixed rule.

#### Pitfall: Following a plan after a step has changed the facts

A team has its agent write a plan and then runs the steps in order,
because the plan looked right. Step three returns a result that makes
step four wrong, such as a stock level of zero, and step four still
runs, reports success and ticks its box. In the `follow` run the trace
shows only `'ok': True` lines and a full list of ticks, and the customer
still got a message that isn't true. Rule: after each step that can change
the facts, give the model the plan and the results, and let it change
the steps that haven't run.

The exercise uses a second goal, "Replace the broken toaster from order
1090.", with the same plan. Its step three returns a fact that
`revise_plan` has no rule for. The `exercise` step prints the plan before
and after the model revisits it.

#### Example: The toaster before the exercise

Run this, and compare what you see with the output below.

```sh
python3 agent.py exercise
```

Output:

```text
before:
plan v1:
  1. [x] Look up order 1090
  2. [x] Read the returns policy
  3. [x] Check that a toaster is in stock
  4. [ ] Send a replacement toaster
  5. [ ] Email the customer that a new toaster has shipped
step 3 returned: {'ok': True, 'item': 'toaster', 'in_stock': 6, 'recalled': True}
after:
  the model kept plan v1
```

## Exercise

The toaster's step three returns `in_stock: 6` and `recalled: True`, and
the model keeps plan v1, so step four would send the customer another
recalled toaster. In your copy of the
`planning` folder, change `revise_plan` so that a recalled item replaces
steps four and five with a refund and an email that names the recall,
and keep the rule for an item that is out of stock. Run
`python3 agent.py exercise`. The output is the plan before and after the
revision. Writing the rule yourself shows how much of the pattern is the
check after each step.

A good result: plan v2 of the toaster has steps one to three ticked, a
refund of order 1090 as step four and an email about the recall as step
five, and `python3 agent.py revise` prints the same kettle plans as
before. Compare with `python3 model_answer.py` in the same folder. Which
other result of step three should change the rest of the plan, and
which step would it change? The opening comment of `model_answer.py`
gives one answer.

Stretch: Give each step an expected result, such as in_stock above 0 for the stock check, and change work so that it calls the model again only when a result differs from what the step expected. Run cost again and compare the model calls.

## Recap

1. Planning has the agent write the steps to a goal before it takes
   them, as a list it works through and ticks off (Planning and reasoning: the loop, plan-then-execute, hierarchy, Agent Engineer Course).
2. A written plan shows a reader where the run is, and in research on
   math problems a plan-first prompt led to fewer answers that skipped a
   step (Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2023), 2609-2634).
3. A plan can go stale when a step returns a new fact. Give the model the
   plan and the results after a step, and let it change the steps still
   to come, at the cost of a model call each time.
4. Before the first step, the plan is text and nothing has changed, so a
   person can correct it cheaply, as Claude Code's plan mode lets you do
   (Choose a permission mode, Claude Code documentation). For a task of one or two steps a plan
   costs a call and gives little.

#### Habit

The next time an agent offers a plan before it starts on a task, read
each step that changes something and correct it before you let it run.

## References

- Planning and reasoning: the loop, plan-then-execute, hierarchy, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/06-planning-and-reasoning/
- Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2023), 2609-2634, https://aclanthology.org/2023.acl-long.147/
- Choose a permission mode, Claude Code documentation, https://code.claude.com/docs/en/permission-modes
- Agentic design patterns: ReAct, reflection, tool use, planning, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/04-agentic-design-patterns/
- Tools reference, Claude Code documentation, https://code.claude.com/docs/en/tools-reference