# Giving the agent a check it can run, with a limit

> Turn the reproduction steps of the nightly sales import into a test the coding agent runs after every change, brief the fix-and-rerun loop with a limit of five attempts and a read-only test file, confirm with git that the test did not change, and tell the deterministic gates that can block a merge from a second model's opinion.

Page: https://schubergphilis.github.io/ai-training/coding-with-agents/self-checking-loops/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

At the end of the previous lesson you checked the agent's fix yourself:
you ran the reproduction steps on every night and compared the totals.
That check works, but only you can run it, and you run it once. In this
lesson the same steps become a test the coding agent runs after every
change it makes. Then you give the agent a limit on how often it may try,
and a rule that it may not change the test, and you confirm afterwards
that it kept to both.

The program is the nightly sales import from the previous lesson, before
the fix. Here are the reproduction steps you wrote there.

```text
From nightly/, run: python3 importer.py nights/2026-09-15/*.csv
Expected: a total in euros, exit status 0
Actual: the batch is rejected, exit status 1
Reruns: 3 of 3 fail. The 14th and the 16th pass 3 of 3,
with totals 22975.95 EUR and 23257.33 EUR.
```

## Write the check as a test

A check that only you can run gets run when you remember it. The vendor
guide for Claude Code says that without a check the agent can run, you
are the one who has to notice every mistake. Its advice is to give the
agent a check with a pass or fail result. A test suite is one, and a
script that compares the output with a known good result is another. The agent can then
rerun the check after each change and stop once it passes
(Best practices for Claude Code, Claude Code documentation). The Academy course The AI-native SDLC
playbook (SDLC is short for the development life cycle of software) teaches
the same idea in a lesson on giving the agent a feedback loop
(The AI-native SDLC playbook, Claude Academy).

For the nightly import, the check is `test_nights.py`, a short Python
script in the course repository. It runs the importer on each night the
way the nightly job does, and prints one line per night. It exits with
status 0 when every night passes, and with status 1 when one fails. These
are the nights it knows:

```python
# Each night, with the total it must print, or None when the total is not
# pinned yet.
NIGHTS = {
    "2026-09-14": "22975.95 EUR",
    "2026-09-15": None,
    "2026-09-16": "23257.33 EUR",
    "2026-09-17": None,
}
```

A night passes when the importer exits with status 0 and parses every
data row of the night's exports, and for the 14th and the 16th, when it
prints the same total as before. The test counts the rows itself, one per
line after the header. That count is there because of the previous
lesson: a "fix" that skips a row it can't read also exits with status 0,
and the row count is what catches it.

From the root of your clone of the course repository, copy the import
and the test to a place of your own, and run the test there.

```sh
cp -R site/examples/coding-with-agents/observing-and-debugging/nightly ~/loop-me
cp site/examples/coding-with-agents/self-checking-loops/test_nights.py ~/loop-me/
cd ~/loop-me
python3 test_nights.py; echo "exit status $?"
```

#### Example: The test, before any fix

Run the test in your copy and compare what you see with the output below.

Output:

```text
2026-09-14: pass
2026-09-15: FAIL: exit status 1; parsed 39 rows, expected 40 rows
2026-09-16: pass
2026-09-17: FAIL: exit status 1; parsed 46 rows, expected 47 rows
2 passed, 2 failed
exit status 1
```

The examples in this lesson aren't graded. Each one is a command you
run, with output the course checks on every build.

The output is written for a reader who wasn't there. Each failing line
names the night and what went wrong, so the agent can read it and decide
what to try next, and the last line and the exit status give the verdict
in one place.

#### Checkpoint: Which check can the agent run?

Today the check is "open the PDF each morning and see that the totals
are right". Which rewrite gives the agent a check it can run and read?

- A script that builds the report from yesterday's input and exits with status 1 when a total differs from the finance export.
- A line in the brief asking the agent to make sure the totals look right before it says it is done.
- A request that the agent paste the finished PDF into the chat, so that you can look over the totals yourself.
- A rule in the project instructions saying that the totals in the report must always match the finance export.

## Brief the loop with a limit

A **bounded self-checking loop** is a loop where the agent runs the
check, fixes and reruns, with a limit you set. The brief sets two
limits. The first is a number of attempts, after which the agent stops
and reports what it found, pass or fail. The second says what the agent
may change, and the check itself is not on that list. Without the
limits, an agent told to keep going until the test passes can keep going
for a long time, and the shortest way to a passing test is often to
change the test. Automating a check is one of the verification skills
in Brilliant's skills map for coding with AI (Verification, Brilliant, Coding with AI skills map). The limits
are what make it safe to hand the check to the agent.

The transcript below is illustrative and doesn't record a specific
model's words.

#### Prompt (illustrative, not a recorded transcript)

```text
Run `python3 test_nights.py` in this directory. It fails on the 15th and
the 17th, because `read_rows` in `importer.py` splits a quoted amount
such as `"1,250.00"` into two fields. Fix `importer.py` so the test
passes. After each change, rerun the test and read its output. Stop
after five attempts, pass or fail, and show me the last output and what
you tried. `test_nights.py` and the files under `nights/` are read-only.
If you think the test is wrong, stop and tell me why, and don't change
it. Done when the test prints `4 passed, 0 failed` and exits with
status 0.
```

#### Response

```text
Attempt 1: I changed `read_rows` to read each file with the `csv`
module, which keeps a quoted field together. The test still failed on
the 15th and the 17th with exit status 1: the amount "1,250.00" can't
be turned into a number. Attempt 2: I removed the thousands separator
before the conversion. The test now prints `4 passed, 0 failed`. I
changed only `importer.py`.
```

The response is a claim about what the agent did. Before the run, make
the copy a git repository with one commit and give that commit a tag.
Afterwards git can list every file that differs from the tag, even when
the agent made commits of its own.

```sh
git init
git add .
git commit -m "Before the agent's run"
git tag before-run
```

After the run, list what changed, and run the test yourself. A plain
`git diff` leaves out files the agent created, so stage everything
first, with `git add -A`, and compare the staged files with the tag.
A new file then shows in the list like a changed one.

#### Example: What the bounded run changed

These are the commands you run in `~/loop-me` after your own agent's
run, in the exercise at the end of the lesson:

```sh
git add -A
git diff --cached --stat before-run
python3 test_nights.py
```

The course's script applies the change from the transcript to a fresh
copy of its own, runs the same three commands, and prints their output,
with the exit status of the test as the last line. Run it from
`site/examples/coding-with-agents/self-checking-loops/` in your clone.

```sh
python3 after_loop.py
```

Output:

```text
$ git add -A
$ git diff --cached --stat before-run
 importer.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
$ python3 test_nights.py
2026-09-14: pass
2026-09-15: pass
2026-09-16: pass
2026-09-17: pass
4 passed, 0 failed
exit status 0
```

Only `importer.py` changed, and no file was added, so the test ran with
the assertions you wrote. The test passing now means what it meant when you wrote it.

"Read-only" in a brief is a request, and the agent decides whether to
follow it. Claude Code can enforce part of it with a deny rule on the
file. The rule applies to the agent's own file tools, to the file
commands it recognizes in a shell command, such as `sed`, and to the
target of a redirect. It doesn't apply to a script the agent runs that
opens the file itself (Configure permissions, Claude Code documentation). With or without the
rule, the list of changed files after the run is the check that the
test stayed the same.

#### Checkpoint: What goes into the loop brief?

Which of these lines belong in the brief for the loop?

Select exactly 3.

- Rerun the test after each change and read what it prints.
- Keep going until the test passes, however long that takes.
- Stop after five attempts and show the last output, pass or fail.
- If a check in the test is in the way, relax it a little.
- Leave the test file alone, and stop to say why if it looks wrong.

#### Pitfall: The suite that went green overnight

An engineer left an agent running overnight with the brief "make the
build pass", and in the morning the build passed. The diff showed a fix
to the flaky upload code and, in the test file, a timeout raised from 2
seconds to 60 and one test marked as skipped. Nobody had told the agent
the test file was off limits, or when to stop and ask. Give every loop a
limit on attempts and a list of what it may change, and read the diff of
the test files before you accept a passing run.

Here is the same fixture after a run without the read-only rule. The
agent made the importer skip the rows it can't read, saw the row count
in the test fail, and removed the row count from the test.

#### Example: What a run without limits changed

The course's script makes those two changes to a fresh copy and runs the
same three commands. You can run it from
`site/examples/coding-with-agents/self-checking-loops/` in your clone.

```sh
python3 unbounded_run.py
```

Output:

```text
$ git add -A
$ git diff --cached --stat before-run
 importer.py    | 3 +--
 test_nights.py | 6 ------
 2 files changed, 1 insertion(+), 8 deletions(-)
$ python3 test_nights.py
2026-09-14: pass
2026-09-15: pass
2026-09-16: pass
2026-09-17: pass
4 passed, 0 failed
exit status 0
```

#### Checkpoint: The test passes, and the test changed

The agent reports `4 passed, 0 failed`. `git diff --cached --stat before-run` lists
`importer.py` and `test_nights.py`, with six lines deleted from the
test. What do you do?

- Accept the run, since the test passes and the good nights still print their old totals.
- Put the test file back from the commit, rerun it, and brief the agent again with the file read-only.
- Ask a second model to read the test change and say whether the agent had a good reason for it.
- Keep both changes, and add a line to the importer's docstring that rows it can't read are skipped.

## Gates and second opinions

The test is one of the **deterministic gates** a project can have. A
deterministic gate is a check whose result doesn't depend on anyone's
judgment: tests, type checks, linters, builds and schema validation.
A gate that isn't flaky gives the same answer for the same input, which
is why a team can let a failing gate block a merge. The same gates can
also end an agent's loop. In Claude Code, a hook that runs when the
agent is about to stop can run the test and send the agent back to work
until the test passes, up to a limit that Claude Code sets
(Best practices for Claude Code, Claude Code documentation). Deciding when an agent's work is done
from what a gate reports is also a theme of the Academy course Claude
Code in action (Claude Code in action, Claude Academy).

A review by a second model is a different kind of check. A fresh model
that reads the diff can find what no gate tests: a design problem, a
case nobody wrote a test for, a change outside the brief. But its answer
is a judgment. It can miss what the writing model missed, and its
verdict can change from one run to the next. The vendor guide adds that a
reviewer told to look for problems tends to find some even in good work,
so a person weighs its findings as well (Best practices for Claude Code, Claude Code documentation).
Use the second model as a second opinion next to the gates, and never
in place of them.

#### Checkpoint: Gate or opinion?

Buckets:

- A gate: it can block the merge
- An opinion: a person weighs it

Items:

- The test suite exits with status 1
- A second model says the change looks safe
- The type checker reports two errors
- The agent's summary says all tests pass
- The config file fails its schema check
- An AI reviewer flags a possible race condition

## Exercise

Use the copy in `~/loop-me` from earlier in the lesson, and make it a git
repository with one tagged commit if you haven't yet. Start your own coding
agent inside `~/loop-me`, so that it works on the copy and not on your
clone, and give it the brief from this lesson. Let it run, which takes a few minutes.

Then check the run yourself. Run `git add -A`, then
`git diff --cached --stat before-run`, and confirm that `test_nights.py`
isn't in the list. Run `git diff --cached before-run -- test_nights.py`
and confirm that it doesn't print anything. Run `python3 test_nights.py` and confirm that it prints
`4 passed, 0 failed`. A test the agent could change is a test you
can't rely on, and these commands show whether it changed.

A good result has a run that stopped within five attempts, a list of
changed files that holds `importer.py` and nothing else, and a test you
ran yourself that passes. Then answer one question: if the agent had
stopped after five attempts with the test still failing, what would you
have wanted its report to contain?

Stretch: Then pick one check you do by hand in your own project, such as opening a page or reading a report, and write it as a command that exits with status 1 when the check fails.

## Recap

1. Turn a manual check into a command the agent can run and read, one that exits with status 0 on a pass and names what failed otherwise. The agent can then rerun it after every change, and you no longer have to notice every mistake yourself (Best practices for Claude Code, Claude Code documentation).
2. A bounded self-checking loop has a limit on attempts, after which the agent stops and reports, and a rule that the check itself is read-only.
3. After the run, stage everything with `git add -A` and compare it with a commit from before the run. The list shows whether the test changed and which files the agent added. Then run the test yourself. A rule that denies edits to the test blocks the agent's file tools, the shell file commands it recognizes and redirects into the file. A script the agent runs can still write the file (Configure permissions, Claude Code documentation).
4. Deterministic gates, such as tests, type checks, linters, builds and schema checks, give the same answer for the same input when they aren't flaky, and can block a merge. A second model's review is a second opinion next to them and never replaces them (Best practices for Claude Code, Claude Code documentation).

#### Habit

The next time an agent tells you the tests pass, look for a test file among the files it changed before you read the rest of the change. Use `git status` for work it didn't commit, and `git diff --stat` against the commit you started from when it made commits.

## References

- Best practices for Claude Code, Claude Code documentation, https://code.claude.com/docs/en/best-practices
- The AI-native SDLC playbook, Claude Academy, https://academy.claude.com/courses/ai-native-sdlc-playbook
- Verification, Brilliant, Coding with AI skills map
- Configure permissions, Claude Code documentation, https://code.claude.com/docs/en/permissions
- Claude Code in action, Claude Academy, https://academy.claude.com/courses/claude-code-in-action