# Tests that prove behavior and docs that stay current

> Break a small pricing module on purpose to find the agent-written tests that mirror the code, rewrite one so it proves the price, keep the tests fast and whole in the agent's loop, and review a change for the documentation it left stale.

Page: https://schubergphilis.github.io/ai-training/coding-with-agents/tests-and-docs-that-hold/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

A coding agent writes tests quickly, and they pass. That tells you the
tests agree with the code, and it doesn't tell you they would notice if
the code were wrong. In this lesson you break a small program on
purpose and run the tests an agent wrote for it. Some of them still
pass, and you rewrite one of them so it checks what the program must
do. You see why a test through the small interface of a module keeps
passing after a correct change inside it. Then you review a change the
agent made to the same program, and find the docs it left out of date.

The program is `shop/`, a pricing module for a small web shop, in the
course repository. It adds up the lines of an order and adds shipping:
4.95 euros (EUR), or nothing for an order of 50.00 EUR or more.

```python
FREE_SHIPPING_FROM = 50.00
SHIPPING = 4.95


def shipping(amount):
    """Return the shipping cost for an order with this subtotal."""
    if amount >= FREE_SHIPPING_FROM:
        return 0.0
    return SHIPPING
```

## Break the code on purpose

**Testing** with an agent means keeping tests that prove what the code
does for its users, and reviewing the tests the agent writes as closely
as the code. Agents produce tests readily, and some of those tests can
pass because they repeat what the code happens to do. The Academy course
The AI-native SDLC playbook (SDLC is short for the development life
cycle of software) argues that when an agent produces most of a team's
code, review and testing become the slow steps (The AI-native SDLC playbook, Claude Academy).
A quick way to review a test is to make the code wrong and see whether
the test notices.

From the root of your clone of the course repository, copy the shop to a
place of your own, and run its tests there.

```sh
cp -R site/examples/coding-with-agents/tests-and-docs-that-hold/shop ~/shop
cd ~/shop
python3 test_pricing.py; echo "exit status $?"
```

#### Example: The agent's tests, before any change

Run the tests in your copy and compare what you see with the output below.

Output:

```text
$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: pass
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: pass
5 passed, 0 failed
exit status 0
```

The examples in this lesson aren't graded. Each one is a command you
run, with output the course checks on every build.

The tests are in `test_pricing.py`. `ORDER` is two items
at 12.50 EUR and one at 20.00 EUR.

```python
ORDER = [(2, 12.50), (1, 20.00)]


def test_subtotal_adds_the_lines():
    assert subtotal(ORDER) == 45.00, subtotal(ORDER)


def test_small_order_pays_shipping():
    assert shipping(10.00) == SHIPPING, shipping(10.00)


def test_large_order_ships_free():
    assert shipping(FREE_SHIPPING_FROM + 10) == 0.0, shipping(FREE_SHIPPING_FROM + 10)


def test_total_adds_shipping():
    expected = subtotal(ORDER) + shipping(subtotal(ORDER))
    assert total(ORDER) == expected, total(ORDER)


def test_total_of_a_known_order():
    assert total(ORDER) == 49.95, total(ORDER)
```

Now make a bug a customer would see on every small order. Open
`pricing.py` in your copy and change `SHIPPING = 4.95` to
`SHIPPING = 5.95`. Before you run the tests again, decide which ones
still pass.

#### Checkpoint: Which tests miss the wrong price?

The constant now holds 5.95 where the shop charges 4.95. Which of the
five tests still pass?

Select exactly 4.

- `test_subtotal_adds_the_lines`
- `test_small_order_pays_shipping`
- `test_large_order_ships_free`
- `test_total_adds_shipping`
- `test_total_of_a_known_order`

#### Example: The tests, with the wrong price

Run the tests in your copy again. The course's script makes the same
change to a fresh copy and runs the tests. You can run it from
`site/examples/coding-with-agents/tests-and-docs-that-hold/` in your
clone.

```sh
python3 break_price.py
```

Output:

```text
$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: pass
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: FAIL: got 50.95
4 passed, 1 failed
exit status 1
```

One test out of five caught a wrong price. Two of the tests that passed
never looked at the price, and that's fine: a test for the subtotal
doesn't need to. The other two mirror the implementation.
`test_small_order_pays_shipping` compares the shipping cost with
`SHIPPING`, the same constant the code returns, so it passes whatever
the constant holds. `test_total_adds_shipping` works out the expected
total by calling the same functions that `total` calls. A test like
this repeats the code, and a test that repeats the code agrees with it
when the code is wrong.

The fix is to write down, in the test, the value the program must
produce, worked out by hand or taken from the spec. Rewrite
`test_small_order_pays_shipping` in your copy so it states the price
the shop charges:

```python
def test_small_order_pays_shipping():
    assert shipping(10.00) == 4.95, shipping(10.00)
```

#### Example: The rewritten test, with the wrong price

Run the tests with the wrong price still in `pricing.py`. The course's
script makes both changes to a fresh copy:

```sh
python3 rewrite_test.py
```

Output:

```text
$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: FAIL: got 5.95
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: FAIL: got 50.95
3 passed, 2 failed
exit status 1
```

The rewritten test now fails, and its line names the wrong value. Put
`SHIPPING = 4.95` back in your copy and run the tests once more: all
five pass again, and the rewritten test now proves the price.

#### Checkpoint: Which rewrite would catch the wrong price?

`test_total_adds_shipping` passed with the wrong price. Which line in
its place would fail while `SHIPPING` holds 5.95?

- `assert total(ORDER) == subtotal(ORDER) + SHIPPING`
- `assert total([(1, 30.00)]) == 34.95`
- `assert total(ORDER) > subtotal(ORDER)`
- `assert total(ORDER) == round(total(ORDER), 2)`

#### Pitfall: The test that learned the new answer

An engineer asked an agent to add discount codes to a checkout. After
the change, a test that checked the total of a known order failed by
one cent, because the new code rounded twice. The agent changed the
assertion to compare the totals to one decimal place, and the suite
went green while every order with a discount showed a total one cent
off. A new expected value in a test is a change to what the program
promises. Review it like a change to the spec, and reject one the brief
didn't ask for.

## Keep the tests fast and unchanged

An agent reruns the tests after every change. How long they take
decides how many attempts fit in a loop: a one-second suite can run
after each edit, and a twenty-minute suite gets run once at the end, or
skipped. The example instructions file in the vendor guide for Claude
Code has a line for this. It asks the agent to prefer running single
tests to the whole suite, for speed (Best practices for Claude Code, Claude Code documentation).
Give the agent the fast tests for the part it changes, and run the full
suite before the merge.

The agent also must not change the tests to get a pass. The shortest
way to make a failing change pass is to delete the test, skip it or
loosen its assertion, as the agent in the pitfall did. When an agent
may edit the tests, the blast radius of one wrong change includes the
check that should catch it. So the rule for an agent is that it never
deletes or weakens a test. A test it thinks is wrong makes it stop and
explain why, and the decision is yours.

A rule in the project instructions is a request. Claude Code reads its
instructions file when a session starts and uses it as context, and
nothing in the file is enforced (How Claude remembers your project, Claude Code documentation). The
rule makes a weakened test less likely, and the review of the test
files is what finds one.

#### Checkpoint: The suite is too slow for the loop

Which way of running the tests fits the agent's loop?

- Let the agent mark the slowest tests as skipped for the run, and turn them back on before the merge.
- Run no tests in the loop, and run the full suite yourself once the agent says it is done.
- Run the fast tests for that module after every change, and run the full suite once before the merge.
- Run the full suite after every change, and drop the limit of five attempts so the agent has time.

#### Checkpoint: The tests pass, and one assertion changed

The brief didn't ask for a change to the total of an order without a
discount code. What do you do?

- Accept the change, since the new assertion still checks that the total is about right.
- Put the old assertion back, run the tests, and brief the agent again, and tell it not to change the existing tests.
- Ask the agent why it changed the test, and accept the change if its reason sounds right.
- Keep the new assertion, and add a comment above it that says why the check was loosened.

## Test through a small interface

**Deep modules** do a lot behind a small interface. John Ousterhout
describes the interface of a module as what its callers have to learn,
and what the module does for them as what they get in return. He argues
that a good module asks little of its callers and does a lot for them
(A Philosophy of Software Design, 2nd Edition, Yaknyam Press, 2021). A shallow module has an interface nearly as large as
its code, so its callers have to know how it works inside.

The shop's interface for a price is `total(lines)`. A caller passes the
lines of an order and gets the amount to pay, without knowing the
threshold, the shipping price or the rounding. `subtotal` and `shipping`
are public as well, but they are helpers that `total` calls. The rewrite
of `test_total_adds_shipping` that catches a wrong price uses only
`total`:

```python
assert total([(1, 30.00)]) == 34.95
```

It fails when the price is wrong, and it keeps passing when an agent
changes the inside of `pricing.py` correctly, for example by storing the
prices in cents. The test you rewrote earlier, `assert shipping(10.00)
== 4.95`, also states the right price, but it calls a helper. A correct
change to what `shipping` takes or returns makes that test change too,
and the `total` test stays as it is.

The two tests that mirrored the code had a different problem: they took
their expected value from the code, so they agreed with a wrong price. A
test through the interface, with a value from the spec, has neither
problem.

A deep module also helps the next session. An agent that changes the
shipping rule edits `pricing.py` and its tests and no other code,
because no caller knows the rule. An agent that only calls the module
needs to read only its interface. Ousterhout's name for the opposite of
hiding a decision is information leakage. When the shipping price is
written into the code of several modules, a change to the price has to
change each of them (A Philosophy of Software Design, 2nd Edition, Yaknyam Press, 2021). An agent can leak a decision in
a change that passes every test, so review the interface in a diff. When
a function gets new parameters, ask what each caller now has to know.
The README and `docs/pricing.md` also state the price, and the next
section reviews them.

#### Checkpoint: The function got two new parameters

An agent was asked to fix how `total` rounds. Its diff changes
`total(lines)` to `total(lines, free_from, shipping_cost)` and edits all
twelve callers to pass `50.00` and `4.95`. The tests pass. What do you
do?

- Approve it, because a function that takes its values as parameters is easier to test with other prices.
- Approve it, because the tests pass and the agent updated all twelve callers in the same change.
- Ask the agent to keep the interface `total(lines)`, with the price and threshold inside the module.
- Ask the agent to document the two new parameters in the docstring, and then approve it.

## Documentation is part of the change

**Documentation** is the written explanation of how a system behaves:
the README, the pages in `docs/`, and the comments and docstrings in
the code. An agent can write good documentation from code, and it can
just as easily change the code and leave the documentation saying
something else. Treat the docs as part of the change. The brief names
what to update, the project instructions say where the docs are, and the
review checks them.

The instructions file of the shop, `AGENTS.md`, says where its docs are.
The Claude Code guide suggests that an instructions file links to
detailed documentation and doesn't copy it in (Best practices for Claude Code, Claude Code documentation).

```text
- Run `python3 test_pricing.py` after every change, and read what it prints.
- Never delete a test or loosen an assertion to make a change pass. If a
  test looks wrong, stop and say why.
- Documentation is in three places: `README.md` for customers,
  `docs/pricing.md` for the full rules, and the docstrings in `pricing.py`.
  A change to a price or a threshold updates all three in the same change.
```

The shop raises its free-shipping threshold. The brief and the reply
below are illustrative and don't record a specific model's words.

#### Prompt (illustrative, not a recorded transcript)

```text
Raise the free-shipping threshold in `pricing.py` from 50.00 EUR to
60.00 EUR, and add a test at the new threshold. Update the documentation
that `AGENTS.md` lists. Run `python3 test_pricing.py` after each change.
Done when the tests pass and none of the places `AGENTS.md` lists still
gives 50.00 EUR as the threshold.
```

#### Response

```text
I changed `FREE_SHIPPING_FROM` to 60.00 and updated the docstring of
`pricing.py`. I added `test_free_shipping_starts_at_sixty`, which checks
60.00 and 59.99. The tests print `6 passed, 0 failed`.
```

The reply lists the docstring and says nothing about the other two
places. You brief your own agent the same way in the exercise at the
end of the lesson. To see what the change touched, make the copy a git
repository with one tagged commit before the run:

```sh
git init
git add .
git commit -m "Before the agent's change"
git tag before-change
```

After the run, stage everything with `git add -A`, so that a file the
agent created shows as well, and list what differs from the tag. Then
search the three places for the old threshold.

#### Example: What the change touched, and what it missed

The course's script makes the agent's change to a fresh copy of the
shop, committed and tagged `before-change` first, and runs the same
commands you run after your own agent's run in the exercise. You can
run it from `site/examples/coding-with-agents/tests-and-docs-that-hold/`
in your clone.

```sh
python3 the_change.py
```

Output:

```text
$ git add -A
$ git diff --cached --stat before-change
 pricing.py      | 4 ++--
 test_pricing.py | 6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)
$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: pass
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: pass
test_free_shipping_starts_at_sixty: pass
6 passed, 0 failed
exit status 0
$ grep -n "50.00" README.md docs/pricing.md pricing.py
README.md:5:Shipping costs 4.95 EUR. Orders of 50.00 EUR or more ship free.
docs/pricing.md:7:the free-shipping threshold, and free from 50.00 EUR. The threshold is
```

The tests pass, and the list of changed files holds neither
`README.md` nor `docs/pricing.md`. The search finds the old threshold
in `README.md` and in `docs/pricing.md`, and nothing in `pricing.py`, whose docstring the
agent did update. A customer who reads the README expects free
shipping on a 55.00 EUR order and pays 4.95 EUR. Send the two stale
lines back to the agent, and review the next diff the same way.

#### Checkpoint: The tests pass. Is the change done?

What do you do before you approve the change?

- Approve it, since the tests pass and the docstring next to the constant was updated as well.
- Ask the agent to confirm that it updated all the documentation, and approve when it says yes.
- Ask the agent to write the documentation again for the whole project, from the code as it is now.
- Search the places the instructions list for the old threshold, and send the stale lines back.

## Exercise

The exercise takes about ten minutes and gives you one rewritten test
and a short list of stale lines. It practices the checks that find a
test that can't fail and documentation that no longer matches the code.

From the root of your clone, make a fresh copy of the shop for the
first part:

```sh
cp -R site/examples/coding-with-agents/tests-and-docs-that-hold/shop ~/shop-boundary
cd ~/shop-boundary
```

In `shipping`, change `>=` to `>`. Before you run the tests, write down
which of the five you expect to fail. Run `python3 test_pricing.py` and
compare. For every test that passes, name one bug in `pricing.py` it
would not catch. Then rewrite `test_large_order_ships_free` so that it
fails with the break in place and passes without it.

For the documentation, go back to the root of your clone, make one more
fresh copy and give it a tagged commit:

```sh
cp -R site/examples/coding-with-agents/tests-and-docs-that-hold/shop ~/shop-docs
cd ~/shop-docs
git init
git add .
git commit -m "Before the agent's change"
git tag before-change
```

Start your own coding agent inside `~/shop-docs`, so it works on the
copy and not on your clone, and give it the brief from this lesson.
When it is done, run these commands:

```sh
git add -A
git diff --cached --stat before-change
python3 test_pricing.py
grep -n "50.00" README.md docs/pricing.md pricing.py
```

A good result has a list of expected failures that matches the run, a
rewritten test that fails with the break and passes without it, and a
review that names every place `AGENTS.md` lists that still gives the old
threshold, or confirms that none does. Then answer one question: which
test in your own project would you break first, and why that one?

Stretch: Then pick one test an agent wrote in your own project, break the line of code it is about on purpose, and see whether the test fails.

## Recap

1. Review an agent's test by making the code wrong on purpose and running it. A test that still passes either gives the code an input that never reaches the broken line, or takes its expected value from the code it tests, so it mirrors the implementation. Rewrite it with an input that reaches the line and the value the program must produce, from the spec or worked out by hand.
2. Give the agent the fast tests for the part it changes to run in its loop (Best practices for Claude Code, Claude Code documentation), and run the full suite before the merge. The agent never deletes, skips or loosens a test to make a change pass. When a test looks wrong, it stops and says why.
3. A rule in the instructions file is context for the agent, and nothing enforces it (How Claude remembers your project, Claude Code documentation). Read the diff of the test files to confirm that no test was deleted, skipped or loosened.
4. Deep modules do a lot behind a small interface (A Philosophy of Software Design, 2nd Edition, Yaknyam Press, 2021). Test them through that interface, and review an agent's diff for a change that makes callers know what the module used to hide.
5. Documentation is part of the change. The brief names what to update, the instructions say where the docs are, and the review searches those places for what the change made stale.

#### Habit

The next time an agent's change comes with new tests, change one line of its code to make it wrong, run the tests, and see which of them notice before you approve.

## References

- The AI-native SDLC playbook, Claude Academy, https://academy.claude.com/courses/ai-native-sdlc-playbook
- Best practices for Claude Code, Claude Code documentation, https://code.claude.com/docs/en/best-practices
- How Claude remembers your project, Claude Code documentation, https://code.claude.com/docs/en/memory
- A Philosophy of Software Design, 2nd Edition, Yaknyam Press, 2021