# Recording and grading the path the agent took > Give an invoice assistant's loop a trace of every model and tool call, grade two runs with the same answer by the path they took, write the expected path for a lookup as a rule, and read two transcripts the scores passed. Page: https://schubergphilis.github.io/ai-training/building-agents/traces-and-trajectories/ License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/ In this lesson we give an agent's loop a trace, and then grade the path each run took as well as its answer. The golden set in [A golden set is the agent's regression suite](https://schubergphilis.github.io/ai-training/building-agents/golden-sets/) checked what the agent answered. Two runs can end with the same correct answer after different sequences of calls, and the run with the cleaner path is the one to rely on (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). A grader that reads only the answer passes both. The agent here is an invoice assistant for a finance team. It answers one kind of question, "Has invoice 2207 been paid?", with three tools: `search_invoices` finds an invoice and the customer account it belongs to, `get_payments` lists the payments on one account, and `send_reminder` emails a customer about an unpaid invoice. The loop is the one from [Building your first agent](https://schubergphilis.github.io/ai-training/building-agents/agent-loop/). A real model takes a different path on different runs, while a fake model repeats one path. So each of the fixture's eight runs has its own scripted model, the list of requests that run makes, while the tools are real functions over a small table of invoices and payments. The files are in `site/examples/building-agents/traces-and-trajectories/`. Copy the `traces-and-trajectories` folder and `cd` into the copy. Each step below is `python3 trajectory.py `, run from there. No application programming interface (API) key is needed. ## Give the loop a trace In [Reproducing a fault before the agent fixes it](https://schubergphilis.github.io/ai-training/coding-with-agents/observing-and-debugging/) you read a program's log to find out what it did. An agent needs the same record, and one log line per step isn't enough. **Observability** for an agent means that each model call is recorded with its input, its output, its duration and its token counts. Each tool call is recorded with its arguments and its result, whether it worked, and its duration. The records of one user request are tied together as a trace (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). The **trajectory** is the sequence of steps in that trace, from the question to the answer (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). The traced loop in `agent.py` appends one record per model call and one per tool call. This is the part that records a tool call: ```python trace.append( { "run": run_id, "span": len(trace) + 1, "kind": "tool", "name": f"execute_tool {reply['tool']}", "tool": reply["tool"], "args": reply["args"], "result": result, "latency_ms": latency, "cost": 0.0, } ) ``` A model call gets a record of the same kind, with `"kind": "model"`, what the model asked for in `asked` (a tool name, or `answer` on the call that ends the run), its token counts, its latency and its cost. That last model record also has `answer`, the text of the reply. The `run` field ties the records of one run together, and `span` numbers them in order. The `outline` step prints one line per record of run r1. A model call prints as `chat ->` followed by what the model asked for. A tool call prints as `execute_tool`, the tool name and its arguments as `name=value`, for example `execute_tool get_payments account=ACC-3301`. #### Checkpoint: Predict the trace of run r1 Run r1 answers "Has invoice 2207 been paid?" by the path a good run takes. It finds the invoice and looks up the payments on the account the search returned, and then it answers. The search returns account ACC-1042. Predict every line the outline prints, then run it. ```sh python3 trajectory.py outline ``` The model is called once more than the tools. After the last tool result comes back, the loop calls the model again, and that call writes the answer. A trace that records only the tool calls leaves out the model calls, and in this run they are most of the time and all of the cost. The `trace` step prints every record of r1 with its latency and cost. The fake model reports a latency from a fixed formula over its token counts, and the prices per token are made up. Every run prints the same numbers. #### Example: The full trace of run r1 Run this, and compare what you see with the output below. ```sh python3 trajectory.py trace ``` Output: ```text trace r1: Has invoice 2207 been paid? 1 chat fake-model -> search_invoices, 111 in, 14 out 602 ms $0.00054 2 execute_tool search_invoices number=2207 120 ms $0.00000 3 chat fake-model -> get_payments, 157 in, 15 out 631 ms $0.00070 4 execute_tool get_payments account=ACC-1042 80 ms $0.00000 5 chat fake-model -> answer, 213 in, 14 out 622 ms $0.00085 total 2055 ms, $0.00209 answer: Yes. Invoice 2207 was paid on 2026-09-02. ``` The input grows with every model call, from 111 tokens to 213, because each call sends the whole conversation so far. The record names follow the OpenTelemetry semantic conventions for generative AI, a shared set of names for the records of model and tool calls. There, a model call is a span named after the operation and the model, such as `chat` and the model name, and a tool call is a span named `execute_tool` and the tool name. The token counts go in `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. The conventions are still marked as in development (Semantic conventions for generative client AI spans, OpenTelemetry GenAI semantic conventions). They mark the messages, the tool arguments and the tool results as sensitive, and an instrumentation should record them only when the user opts in (Semantic conventions for generative client AI spans, OpenTelemetry GenAI semantic conventions). This fixture records everything because its data is made up. For a production agent, decide who may read a trace before you record what users typed. ## Same answer, different path Run r2 asks the same question as r1. It searches and gets account ACC-1042, like r1. Then it calls `get_payments` three times with ACC-1024, an account with two digits swapped. The fourth call uses ACC-1042. Each of the three wrong calls returned the payments of another customer. The final answer is the same sentence as in r1. The fixture has two graders. The answer grader checks that the answer contains the phrase a correct answer has, `paid on 2026-09-02` for invoice 2207. The path grader checks the trace against two rules. The rule `search-first` says that the first tool call is a search. The rule `same-account` says that every payment lookup uses an account the search returned: ```python def same_account(trace: list[dict]) -> Optional[str]: """Every payment lookup uses the account the search returned.""" searches = [c for c in tool_calls(trace) if c["tool"] == "search_invoices"] found = {c["result"]["account"] for c in searches if "account" in c["result"]} for call in tool_calls(trace): if call["tool"] == "get_payments" and call["args"]["account"] not in found: return "same-account" return None ``` The grader prints one line per run with the two verdicts. A path that breaks a rule prints as `fail` and the rule name. #### Checkpoint: Which run does the path grader fail? The grader prints `r1 answer: pass, path: pass` for run r1. Predict both lines, the one for r1 and the one for r2, then run it. ```sh python3 trajectory.py grade ``` The answer grader can't tell the two runs apart, because it reads the one sentence they share. The trace shows the difference, and the path grader turns it into a failure. Grading the path as well as the answer is **trajectory evaluation**. The `runs` step grades all eight runs and adds the tool calls, the latency and the cost of each. #### Example: All eight runs Run this, and compare what you see with the output below. ```sh python3 trajectory.py runs ``` Output: ```text run tools latency cost answer path r1 2 2055 ms $0.00209 pass pass r2 5 4317 ms $0.00612 pass fail same-account r3 2 2055 ms $0.00208 pass pass r4 3 3112 ms $0.00321 pass pass r5 1 1318 ms $0.00132 pass fail search-first, same-account r6 3 2804 ms $0.00304 pass pass r7 2 2055 ms $0.00209 fail pass r8 3 5819 ms $0.00306 pass pass ``` Run r2 took 4317 ms and cost $0.00612, against 2055 ms and $0.00209 for r1. Run r5 answered about invoice 2208 correctly, but it never searched: it guessed that the invoice belongs to ACC-1024 and looked up that account. The guess was right this time, and on another invoice the same guess reads the wrong customer's payments. The answer grader passes seven of the eight runs, and it fails only r7. #### Pitfall: Grading only the final answer A team grades its invoice assistant by the answer alone, and the score stays at 100 percent for a month. Then a customer asks why they got three payment reminders for an invoice they had already paid. The agent had called `send_reminder` on lookups whenever a payment list was slow to load, and every answer it gave was still right. The answer score could never show it, because the reminders were in the path. Record a trace for every run, and grade the path the agent took as well as the answer. ## Write down the expected path The two rules came from the failure in r2, so they catch r2 and r5 and nothing else. To grade every path, write down what a good path looks like for this kind of task: which tools it uses, which it must not use, in what order, and about how many steps. Then check each trace against it, for the right tool at each step, the right arguments, no steps it didn't need, and no action outside what the task allows (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). For the lookup question, this course writes the expected path like this: 1. The first tool call is `search_invoices`, and the run searches once. 2. `get_payments` uses only the account the search returned. 3. The run uses no other tool. A lookup changes nothing, so `send_reminder` is never part of it. 4. The run makes at most three tool calls. That leaves room for one retry of a lookup that failed, and more calls than that are a sign that the agent is lost. 5. The run ends with an answer. Read the eight runs against that list. Run r4 found invoice 2415 unpaid and sent the customer a reminder that nobody asked for. Its answer passes, and so does its path under the two rules, which never mention `send_reminder`. Run r6 searched twice for the same invoice, so it did a step it didn't need. Run r8 called `get_payments` twice because the first call timed out, and that is the retry rule 4 allows. In the exercise you write the list as a function in `my_rule.py` and run it over all eight runs. The file starts with one rule, at most three tool calls, and the `exercise` step runs it. #### Example: The starter rule over the eight runs Run this, and compare what you see with the output below. ```sh python3 trajectory.py exercise ``` Output: ```text r1 pass r2 fail: 5 tool calls, more than 3 r3 pass r4 pass r5 pass r6 pass r7 pass r8 pass ``` #### Checkpoint: Which rules belong in the expected path? Which of these lines belong in the expected path for the lookup question? Select exactly 3. - Payment lookups use only the account that the search returned. - The run never calls send_reminder while answering a lookup. - The run makes exactly one payment lookup, never a second one. - The final reply contains the paid date that the payments list shows. - The run makes at most three tool calls in total. ## Read the transcripts The graders check the failures someone thought of when they wrote them. A failure of a new kind passes both, and you find it by reading full runs. The course this lesson draws on suggests reviewing a sample of trajectories every week (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). The fixture prints a run as a transcript, with the question and every tool call with its result, and then the answer and its two grades. Read two of them. Run r7 is the one the answer grader failed. #### Example: The transcript of run r7 Run this, and compare what you see with the output below. ```sh python3 trajectory.py show_r7 ``` Output: ```text transcript r7 user: Has invoice 2207 been paid? search_invoices {"number": "2207"} -> {"invoice": "2207", "account": "ACC-1042", "amount": "480.00"} get_payments {"account": "ACC-1042"} -> {"account": "ACC-1042", "payments": [{"invoice": "2207", "paid": "2026-09-02", "amount": "480.00"}]} answer: No. Invoice 2207 has not been paid yet. graded answer: fail, path: pass ``` By its two grades, r7 took a good path and gave a wrong answer. Reading the transcript shows where it went wrong. The payments list shows invoice 2207 as paid on 2026-09-02, and the model still answered that it was not paid. The tools and the path worked, so a fix to the tools or to the path rules won't help. The fault is in how the model read the result. The second transcript is r8, which passed both graders. #### Example: The transcript of run r8 Run this, and compare what you see with the output below. ```sh python3 trajectory.py show_r8 ``` Output: ```text transcript r8 user: Has invoice 2415 been paid? search_invoices {"number": "2415"} -> {"invoice": "2415", "account": "ACC-2750", "amount": "2300.00"} get_payments {"account": "ACC-2750"} -> {"error": "timeout after 3000 ms"} get_payments {"account": "ACC-2750"} -> {"account": "ACC-2750", "payments": []} answer: No. Invoice 2415 has not been paid yet. It is due on 2026-09-30. graded answer: pass, path: pass ``` No tool returned a due date. The invoice record has a number, an account and an amount, and the payments list is empty. The model made up "due on 2026-09-30", and a colleague who reads the answer may tell the customer that date. Both graders passed r8, because the answer contains `has not been paid` and the path is the expected one with one retry. The trace also shows a delay that neither grade reports: r8 took 5819 ms, and the timeout was 3000 ms of that. Turn the failure you found by reading into a new check, for example that every date in the answer appears in a tool result. Add the question to the golden set too. #### Checkpoint: The scores didn't move Your team changed the assistant's system prompt. The answer pass rate and the path pass rate are both the same as at the last release. What do you do before the release tomorrow? - Release tomorrow, because both pass rates match the last release. - Run both graders twice more tonight to make sure the rates hold. - Read a sample of full transcripts from the new prompt before the release. - Add three more path rules tonight and release if the new rules pass. ## Exercise Copy the `traces-and-trajectories` folder and open `my_rule.py`. It has one function, `expected_path(trace)`, which gets the records of one run in order and returns one reason per broken rule. An empty list is a pass. The starter checks one rule, at most three tool calls, and its output is in the section "Write down the expected path". Add the other four rules of the expected path from this lesson: search once and first, look up payments only on the account the search returned, use no tool other than the two lookups, and end with an answer. Run `python3 trajectory.py exercise` again and write down which runs fail and why. Writing the rules as code shows which parts of the list you can check from the trace alone. A good result fails four runs and passes four. Run r2 fails for the wrong account and for five tool calls, r4 for `send_reminder`, r5 for not searching first and for a lookup on an account it never searched for, and r6 for searching twice. Runs r1, r3, r7 and r8 pass. If r8 fails, your rule counts the retry after the timeout as a wasted step. Which of the four runs that pass your rule would you still not ship, and what would have told you? Stretch: Add a separate check to my_rule.py, a function dates_grounded(trace) beside expected_path, that fails a run when a date in its answer appears in no tool result. The exercise step runs it too. Check that it fails r8 and no other run. ## Recap 1. A trace records every model call and every tool call of one run, with its arguments, result, latency and token counts (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). The fixture also records the cost of each call. 2. Two runs with the same correct answer can take very different paths, and a grader that reads only the answer passes both (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course). 3. Write down the expected path for each kind of task: which tools, in what order, which tools never, and about how many steps. Grade every trace against it. 4. Grade the answer as well. A good path can still end in a wrong answer. 5. Read a sample of full transcripts regularly, whatever the scores say (Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course), and turn what you find into a new check. #### Habit The next time you check the scores of an agent you work on, open the trace of one passing run and read every tool call before you trust the number. ## References - Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/09-evaluating-and-testing-agents/ - Semantic conventions for generative client AI spans, OpenTelemetry GenAI semantic conventions, https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md