# Reproducing a fault before the agent fixes it

> Turn a "sometimes fails" report against a nightly sales import into a rerun that fails every time, read the log of a failing run next to a good one to rule out the suspected cause, halve the exports until one row is left, and only then brief the agent with the confirmed cause and check the fix against the same reproduction.

Page: https://schubergphilis.github.io/ai-training/coding-with-agents/observing-and-debugging/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson you get a bug report that says "sometimes fails", and you
turn it into a confirmed cause before a coding agent changes a line. The
program is a small nightly import that adds up the sales exports of eight
stores. The course keeps it as a fixture, so everyone debugs the same
code and the same data. Here is the report.

```text
The nightly sales import failed on Tuesday the 15th and on Thursday the
17th, and ran fine on the 14th and the 16th. Nobody changed the code. I
think the exchange-rate download is sometimes late at night and leaves a
half-written rates.json. Can you ask the agent to make the download more
reliable?
```

The report contains a symptom, two dates, and a theory. An agent handed
this report gets a theory to work from and a request for a fix, and the
fix it writes is aimed at the theory. Asking for a reproduced, explained
cause before any patch keeps the fix from becoming a second bug
(Verification, Brilliant, Coding with AI skills map).

## Reproduce it first

"Sometimes" means you don't yet know what the failure depends on. The
first step is to run the failing case again, the same way, and see
whether it fails every time. The fixture keeps each night's exports in
`nightly/nights/<date>/`, one comma-separated values (CSV) file per
store, and the nightly job runs
`python3 importer.py nights/<date>/*.csv` from `nightly/`. The course's
script runs that command three times for each of the four nights and
prints the result of each run, with the total for a night that passes.

#### Example: Run each night three times

In a clone of the course repository, go to the fixture directory,
`site/examples/coding-with-agents/observing-and-debugging/`, and run the
reproduction script there. The scripts in this lesson all run from that
directory.

```sh
python3 reproduce.py
```

Output:

```text
2026-09-14: ok ok ok (total: 22975.95 EUR)
2026-09-15: rejected rejected rejected
2026-09-16: ok ok ok (total: 23257.33 EUR)
2026-09-17: rejected rejected rejected
```

The examples in this lesson aren't graded. Each one is a command you
run, with output the course checks on every build. The checkpoints ask
what you do with what the commands show.

Every rerun gives the same result as the night it reruns. The failure
now has a precise condition: it happens every time, on the exports of
the 15th and the 17th. Note the totals of the two good nights as well,
because the fix must not change them. Write the steps down now, in a
form someone else can follow, because they're the test the fix must pass
at the end.

```text
From nightly/, run: python3 importer.py nights/2026-09-15/*.csv
Expected: a total in euros, exit status 0
Actual: the batch is rejected, exit status 1
Reruns: 3 of 3 fail. The 14th and the 16th pass 3 of 3,
with totals 22975.95 EUR and 23257.33 EUR.
```

#### Checkpoint: What do the reruns tell you?

The 15th and the 17th fail three runs of three, and the 14th and the
16th pass three runs of three. What does that tell you about the cause?

- That the rates file was half-written on the 15th and the 17th and still is today.
- That the failure follows the exports of the 15th and the 17th, because the same input fails on every run.
- That the failure follows the time of day, and the reruns happened to land on a calm moment.
- That nothing is known yet, because a fault that fails only sometimes can never be made to fail on demand.

## Read what the system recorded

Observing a running system means looking at what the program did, and
not only at what its code says it should do: the log it writes, the
files it reads, the output and the exit status. The code tells you what
should happen, and the log tells you what did (Verification, Brilliant, Coding with AI skills map). The
importer writes one log line per step. Put the log of a failing night
next to the log of a good one.

#### Example: The log of a good night and a failing one

From the fixture directory, the course's script runs the importer on
the 14th and on the 15th, from `nightly/`, and prints each log with its
exit status. To run the two commands yourself instead, `cd nightly`
first, follow each command with `echo $?`, and `cd ..` when you are
done. The next steps start from the fixture directory again.

```sh
python3 night_log.py
```

Output:

```text
$ python3 importer.py nights/2026-09-14/*.csv
INFO reading 8 files
INFO rates: 3 currencies loaded
INFO parsed 46 rows
INFO total: 22975.95 EUR
exit status 0
$ python3 importer.py nights/2026-09-15/*.csv
INFO reading 8 files
INFO rates: 3 currencies loaded
INFO parsed 39 rows
ERROR 1 row(s) with the wrong number of fields, batch rejected
exit status 1
```

Read the two logs line by line. Both read eight files, and both load
three currencies from the rates file. The failing run gets past the
rates step and stops at the next one, where one row has the wrong number
of fields. The rerun rules out the colleague's theory: the rates load in
full, and the batch is still rejected at the next step. Write
that down too, with the evidence, so nobody has to rule it out again.

The log doesn't say which of the eight files holds the bad row. It
narrowed the cause from the whole program to one step, and the step
still has eight suspects.

#### Pitfall: The handler the request never reached

A checkout page sometimes shows an error, and the coding agent starts to
rewrite the order handler with more checks for a field that could be
missing. The engineer opens the service's request log first, and the
failing requests aren't in it. A proxy in front of the service rejected
them for headers that were too large, so the handler never saw them. The
rewrite would have passed its tests and changed nothing for the user. Before
anyone changes the code a theory points at, check in the log that the
failing run reached that code.

#### Checkpoint: The agent proposes retries

The agent says: "The rate download is unreliable at night. I'll add
three retries with a pause between them, and check that `rates.json` is
complete before it is used." The log of the rerun of the 15th is open
next to it. What do you do?

- Let it add the retries, since a more reliable download is useful even if it isn't this bug.
- Ask it to add detailed logging to the download, so the next failing night shows what the download did.
- Ask it to rewrite the import to be more tolerant, so that one bad row no longer rejects the batch.
- Stop it and point at the log of the rerun, where the rates load in full and the parsing step fails after them.

## Halve what is left

Isolating a fault means narrowing it down to its cause in steps, where
each step changes one thing and gives one observation. With eight files
to suspect, the fastest way is to halve them. Run the importer on half
of the files. If that half passes, the bad row is in the other half, and
if it fails, the bad row is in this one. Each run rules out half of what
is left, so eight files take three runs, and a fourth run confirms the
last one. The shell pattern `store-0[1-4].csv` picks four files by name.

```sh
cd nightly
python3 importer.py nights/2026-09-15/store-0[1-4].csv
python3 importer.py nights/2026-09-15/store-0[5-6].csv
python3 importer.py nights/2026-09-15/store-05.csv
python3 importer.py nights/2026-09-15/store-06.csv
cd ..
```

#### Example: Halve the files of the 15th

From the fixture directory, the course's script runs the four commands
above from `nightly/` and prints `ok` or `rejected` for each.

```sh
python3 halve.py
```

Output:

```text
store-01 to store-04: ok
store-05 and store-06: rejected
store-05: ok
store-06: rejected
```

Keep a note of what each run ruled out, one line per run: stores 1 to 4
are ruled out, stores 7 and 8 are ruled out, store 5 is ruled out, and
store 6 fails alone. Halving works best when the failure has one cause.
If both halves had failed, the note would show it at once, and you would
halve each of them.

Now store-06.csv is short enough to read.

```text
date,store,currency,amount
2026-09-15,store-06,GBP,449.30
2026-09-15,store-06,GBP,642.20
2026-09-15,store-06,GBP,"1,250.00"
2026-09-15,store-06,GBP,432.53
```

The fourth line has an amount of a thousand or more, which the store's
system writes in quotes and with a comma. Read `read_rows` in
`importer.py` with that line in mind: it splits each line on every
comma, so `"1,250.00"` becomes two fields and the row has five. This
cause explains the whole report: the nights that failed are the nights
with a big sale, and nothing about the timing matters. The condition is
now precise. The import fails when an export holds an amount of 1,000 or
more.

The same halving works on commits as well as on files. When a fault
appeared between a version that worked and one that doesn't, `git bisect`
does a binary search over the commits between them. It checks out one
commit at a time, you mark it good or bad, and it stops at the commit
that brought the fault in (git-bisect, Git reference documentation).

#### Checkpoint: Which run comes next?

On the 15th, `store-0[1-4].csv` passes on its own. Which run do you do
next?

- The importer on store-05 and store-06 only, because the bad row is now among stores 5 to 8, and half of that is two files.
- The importer on stores 1 to 4 once more, because one passing run of a failure that seemed random proves nothing.
- The importer on store-01 alone, because the first file is where a bad row most often turns up in a batch.
- The importer on stores 5 to 8 with a change that makes it skip bad rows, to see whether the total then looks right.

## Brief the agent with the cause

Only now does the agent get the task. The brief contains everything the
reproduction and the halving found: the command that fails, the file and
the line, the reason, what was ruled out, and what done means. The
transcript is illustrative and doesn't record a specific model's words.

#### Prompt (illustrative, not a recorded transcript)

```text
The nightly import in `nightly/importer.py` rejects the batch for
2026-09-15 every time. To reproduce it, run
`python3 importer.py nights/2026-09-15/*.csv` from `nightly/`. It logs
`1 row(s) with the wrong number of fields` and exits with status 1. The
cause is line 4 of `nights/2026-09-15/store-06.csv`, the amount
`"1,250.00"`: `read_rows` splits on every comma, so the quoted amount
becomes two fields. The rates download is ruled out, because a rerun
with a complete `rates.json` fails the same way. Fix the parsing so that
a quoted amount with a thousands separator is read as one amount, and
change nothing else. The 17th fails with the same log line. Done when
all four nights under `nights/` exit with status 0, and the 14th and the
16th still print 22975.95 EUR and 23257.33 EUR.
```

#### Response

```text
I changed `read_rows` to read each file with the `csv` module, which
keeps a quoted field together, and removed the comma from the amount
before it is turned into a number. Nothing else changed. All four nights
now pass.
```

The response is a claim, and the reproduction steps are the check. Run
them again yourself, on every night, and compare the totals of the
nights that already passed. The course's script applies the same fix to
a copy of `nightly/` and reruns the reproduction.

#### Example: The reproduction, after the fix

From the fixture directory, run the script that applies the fix on a
copy and prints each night's result and total.

```sh
python3 after_fix.py
```

Output:

```text
2026-09-14: ok, total: 22975.95 EUR
2026-09-15: ok, total: 18505.15 EUR
2026-09-16: ok, total: 23257.33 EUR
2026-09-17: ok, total: 23213.13 EUR
```

The two failing nights pass, and the two good nights print the same
totals as before. The fix is verified against the steps that showed the
fault, which is a stronger check than the agent's summary of it.

#### Checkpoint: What goes into the brief?

Which of these belong in the brief you give the agent?

Select exactly 3.

- The command that fails and the output it gives now.
- The colleague's theory about the rates download, as a second thing to fix.
- The file and the line with the quoted amount, and why a split on commas breaks it.
- Permission to change whatever the agent needs to change until the import passes.
- All four nights pass, and the totals of the good nights stay the same.

## Exercise

From the root of your clone of the course repository, copy the
fixture's `nightly/` directory to a place of your own, and work only in
the copy.

```sh
cp -R site/examples/coding-with-agents/observing-and-debugging/nightly ~/debug-me
cd ~/debug-me
```

Take the 17th, which the lesson left alone, through the same steps. Spend
about ten minutes. Write the reproduction steps, with how many reruns
failed. Compare its log with a good night's and write one line on what
the log rules out. Halve the eight files, one run at a time, and keep a
line per run for what it ruled out. Then read the file that is left.
This is the order that keeps a guessed fix out of your code, and it
works the same for a bug in your own project.

A good result has a reproduction that fails three runs of three, a note
that the rerun of the 17th loads the rates in full, and a halving that
ends at `store-03.csv`, where line 4 holds the amount `"2,180.40"`. The
cause is the same as on the 15th. Then answer one question: if you had
briefed the agent with the colleague's report as it was, what would it
have changed?

Stretch: Then start your own coding agent inside `~/debug-me`, brief it with what you found, and verify its fix against your reproduction steps. Compare its diff with the fix the lesson describes.

## Recap

1. Reproduce the fault before anyone changes code. Rerun the failing input and write the steps down, and "sometimes fails" becomes a precise condition and the test the fix has to pass (Verification, Brilliant, Coding with AI skills map).
2. Observing a running system means reading what it recorded, the log, the output and the exit status, next to a run that worked. The log shows how far the failing run got, and a theory about a step the run completed is ruled out.
3. Isolating a fault means halving the suspects: one change, one observation, and a note of what was ruled out. The same halving over commits is `git bisect` (git-bisect, Git reference documentation).
4. Brief the agent with the reproduced, explained cause and a done-criterion from the reproduction, and verify the fix by running those steps yourself.

#### Habit

The next time a bug report says a thing sometimes fails, rerun the failing case three times and write down the steps before you open the code or ask an agent for a fix.

## References

- Verification, Brilliant, Coding with AI skills map
- git-bisect, Git reference documentation, https://git-scm.com/docs/git-bisect