# Cost, caching and pinning the model

> Give an agent loop a cost record and a token budget, cache the prompt prefix that never changes, retry the provider errors that go away after a wait and report the others, and pin the model id.

Page: https://schubergphilis.github.io/ai-training/building-agents/cost-and-provider-operations/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson we give an agent loop a bill. The loop is the one from
[Building your first agent](https://schubergphilis.github.io/ai-training/building-agents/agent-loop/): send the
message list to the model, run the tool it asks for, append the result,
and repeat until it answers. This time every call to the model goes
through a provider that counts tokens and charges for them. We add a cost
record per run and a token budget that stops a run. Then we cache the
part of the prompt that never changes, and retry a call that the provider
turns down with a rate limit. The last section pins the model, so that
its behavior changes only on a date you choose.

The provider is fake. It runs inside the program and never reaches the
network, and no application programming interface (API) key is needed.
Its token counter treats every word and every punctuation mark as one
token, which is a rough stand-in for a real tokenizer. The prices are made
up, and the cache prices follow the Claude API's rule for most models,
which is described below. The files are in
`site/examples/building-agents/cost-and-provider-operations/`. Copy the
folder, and run each step from inside the copy as
`python3 ops.py <step>`.

## What one run costs

The agent is a help desk assistant for a small office, with three tools:
`get_ticket`, `search_kb` for the knowledge base and `read_log` for last
night's log of an office service. Task `t1` is a question about a laptop
that will not start. The run makes three calls: the model asks for the
ticket, then for the knowledge base article, and then it answers.

A real provider reports the tokens of each call in its response. The
Claude API returns `input_tokens` and `output_tokens`, and two more
fields for the prompt cache, `cache_creation_input_tokens` and
`cache_read_input_tokens` (Prompt caching, Claude Platform documentation). The price is per
million tokens, output tokens cost several times as much as input
tokens, and the tool definitions you send count as input
(Pricing, Claude Platform documentation). The fake provider returns the same four fields,
and one function turns them into dollars:

```python
PRICE_INPUT = 3.00  # dollars per million tokens, made up
PRICE_OUTPUT = 15.00
PRICE_CACHE_WRITE = PRICE_INPUT * 1.25
PRICE_CACHE_READ = PRICE_INPUT * 0.1

def dollars(usage: dict) -> float:
    return (
        usage["input_tokens"] * PRICE_INPUT
        + usage["cache_creation_input_tokens"] * PRICE_CACHE_WRITE
        + usage["cache_read_input_tokens"] * PRICE_CACHE_READ
        + usage["output_tokens"] * PRICE_OUTPUT
    ) / 1_000_000
```

The loop keeps a `CostRecord` per run, with the number of calls, the
tokens and the dollars, and adds each response's usage to it. With
`log=print` it prints one line per call.

#### Example: The bill of one run

Run this, and compare what you see with the output below.

```sh
python3 ops.py cost
```

Output:

```text
call 1: input 260, output 23, $0.001125
call 2: input 353, output 26, $0.001449
call 3: input 443, output 28, $0.001749
stop: end_turn
answer: Ticket 311 is open. Hold the power button for 15 seconds, then charge it for 10 minutes.
calls: 3
tokens: 1133
cost: $0.004323
```

The input grows with every call. Call 2 sends everything call 1 sent, plus
the model's tool request and the ticket, and call 3 adds the article. A
loop pays for its whole history again on every turn, so the number of
turns and the size of each tool result decide the bill. The system prompt
and the tool list are 226 of the 260 input tokens of call 1, and they are
in every call. A run costs less than half a cent here, and the number
that matters is that cost times the runs per day. Track it per run and
per task type, and alert on it, so a change in cost shows up the day it
happens (From prototype to production: eval-gated deploys, rollout, cost, Agent Engineer Course).

## A budget that stops the run

A step limit catches a loop that repeats too often, and
[Stopping the loop on purpose](https://schubergphilis.github.io/ai-training/building-agents/stopping-the-loop/) adds
a character budget next to it. When a run has few turns and one of them
is large, such as a tool result much longer than expected, only a token
budget stops it. The Agent Engineer Course sets a hard token cap per
session for this reason (From prototype to production: eval-gated deploys, rollout, cost, Agent Engineer Course). The loop checks the budget
before each call, and when the record has reached it, the run stops with
its own reason:

```python
for _ in range(max_steps):
    if budget is not None and record.tokens >= budget:
        return outcome("budget_spent", None)
    try:
        response = call_provider(provider, model, messages, cache, no_sleep, log, record.calls + 1)
    except ProviderError as err:
        return outcome("provider_error", None, f"{err.status} {err.kind}: {err.message}")
    record.add(response["usage"])
    reply = response["reply"]
    if "answer" in reply:
        return outcome("end_turn", reply["answer"])
    ...
```

`outcome` returns a dict with the stop reason, the answer, and the
record's calls, tokens and cost, in that order. The cost is a string with
a dollar sign and six decimals, such as `'$0.004323'`. `call_provider`
and the `provider_error` stop are the subject of a later section.
`record.tokens` counts every token of every call, input and output.

#### Checkpoint: What does the budget stop return?

Task `t1` runs again, now with `budget=600`. Use the per-call numbers
from the run above, and predict what this prints.

```python
print(run("t1", budget=600))
```

Then run it:

```sh
python3 ops.py budget
```

The run spent 662 tokens, more than the budget of 600, because the check
runs before a call and a call can't be stopped halfway. Set the budget
with that in mind. The caller gets the stop reason, and the tokens and
the cost it has already paid for, and can decide what to tell the user.

## Caching the part that never changes

The system prompt and the tool list are the same on every call of every
run. Prompt caching lets the provider process that prefix once and read
it back on later calls. On the Claude API the cache follows the order of
the request, the tools first, then the system prompt, then the messages.
A cache hit needs an exact match of everything up to the cache breakpoint,
and a cached prefix lasts five minutes by default
(Prompt caching, Claude Platform documentation). For most models, the price of writing the
prefix to the cache is 1.25 times the input price, and the price of
reading it back is 0.1 times the input price (Pricing, Claude Platform documentation). The
API caches a prefix only above a minimum length, which depends on the
model (Prompt caching, Claude Platform documentation).

The fake provider caches the system prompt and the tool list together
when a run passes `cache=True`. The first call writes them to the cache,
and later calls read them.

#### Example: The same run with the prefix cached

Run this, and compare what you see with the output below.

```sh
python3 ops.py cache
```

Output:

```text
call 1: input 34, cache write 226, output 23, $0.001295
call 2: input 127, cache read 226, output 26, $0.000839
call 3: input 217, cache read 226, output 28, $0.001139
stop: end_turn
answer: Ticket 311 is open. Hold the power button for 15 seconds, then charge it for 10 minutes.
calls: 3
tokens: 1133
cost: $0.003272
```

Compared with the first run, call 1 costs more by the extra price of the
cache write, while calls 2 and 3 cost less, and the run drops from
$0.004323 to $0.003272. The token count stays 1133: caching changes the
price of the tokens, and the model still reads all of them. The more
calls share one prefix, the more a run saves. With the Claude API, a run
that starts while the prefix is still in the cache reads it on its first
call as well. Tokens read from the cache also don't count toward the
input-tokens-per-minute rate limit for most Claude models
(Rate limits, Claude Platform documentation).

#### Checkpoint: The time in the system prompt

A teammate adds the current time, to the second, at the start of the
system prompt, so that the model knows when it answers. Caching is on.
What happens to the cost of the prefix?

- Every call writes the prefix to the cache again, so the prefix costs more than it did without caching.
- Only the first call of each run writes the prefix, and later calls in the run read it back.
- The fake provider still reads the prefix from the cache, because it looks the prefix up by model ID.
- The calls fail, because a cached prefix must not change between calls of one run.

## Calling the provider reliably

A provider refuses calls, and the loop must know which refusals to wait
out. The Claude API names its errors by Hypertext Transfer Protocol
(HTTP) status and type. Retry a 429 `rate_limit_error`, a 500
`api_error`, a 504 `timeout_error` and a 529 `overloaded_error` after a
wait (Claude API errors, Claude Platform documentation). A 429 for a rate limit comes with a
`retry-after` header that says how many seconds to wait
(Rate limits, Claude Platform documentation). Report the rest to the caller: a 400
`invalid_request_error` or a 401 `authentication_error` fails again on
every retry. So does a 429 that says the monthly spend cap is reached,
which has no `retry-after` header (Claude API errors, Claude Platform documentation). The official
software development kits (SDKs) already retry transient failures twice
by default, with exponential backoff, and honor `retry-after`
(Claude API errors, Claude Platform documentation). A loop you write yourself needs the same rules.

The fixture's `call_provider` wraps every call. It passes a timeout, so a
call that hangs becomes an error the loop can handle. A timeout or a
retriable status leads to a retry. Before the retry the loop waits for
the `retry-after` seconds when the provider sends them, or for 1, 2 and
then 4 seconds when it doesn't. Any other error goes back to the caller.

```python
RETRIABLE = {429, 500, 504, 529}

def call_provider(provider, model, messages, cache, sleep, log, number, max_retries=3, timeout=30.0):
    attempt = 1
    while True:
        try:
            return provider.create(model, SYSTEM, TOOL_LIST, messages, cache=cache, timeout=timeout)
        except CallTimeoutError as err:
            status, text, wait_for = "timeout", str(err), None
        except ProviderError as err:
            if err.status not in RETRIABLE:
                raise
            status, text, wait_for = f"{err.status} {err.kind}", err.message, err.retry_after
        if attempt > max_retries:
            raise ProviderError(0, "gave_up", f"{status} after {attempt} attempts")
        wait = float(wait_for) if wait_for is not None else 1.0 * 2 ** (attempt - 1)
        log(f"call {number}: {status}: {text}; wait {wait:.1f}s and retry")
        sleep(wait)
        attempt += 1
```

`run` turns a `ProviderError` into the stop reason `provider_error`, with
the error in the result. The fixture never really waits: its `sleep`
does nothing, and the log line says how long a real loop would wait. The
fake provider bills nothing for a call it refuses.

#### Checkpoint: A rate limit on the third call

The fake provider now refuses the third call once. The step prints only
the retry lines of the log, then the calls, tokens and cost of the run.
Predict what it prints. The run without errors made 3 calls for 1133
tokens and $0.004323.

```python
limited = Provider(
    refuse={3: ProviderError(429, "rate_limit_error", "rate limit exceeded", retry_after=2)}
)
result = run("t1", provider=limited, log=retries_only)
print(f"calls: {result['calls']}, tokens: {result['tokens']}, cost: {result['cost']}")
```

Then run it:

```sh
python3 ops.py rate_limit
```

The run ends with the same calls, tokens and cost as the run without the
error, and the log has one retry line. A person reading the log sees that
the limit was hit, and the user sees nothing. The `errors` step shows the
other two paths. The second call times out once and is retried after the
default wait of one second. Then the provider refuses the third call with
a 400, and the loop stops at once.

#### Example: A timeout, then a terminal error

Run this, and compare what you see with the output below.

```sh
python3 ops.py errors
```

Output:

```text
call 1: input 260, output 23, $0.001125
call 2: timeout: no reply within 30s; wait 1.0s and retry
call 2: input 353, output 26, $0.001449
stop: provider_error
answer: None
calls: 2
tokens: 662
cost: $0.002574
error: 400 invalid_request_error: messages: bad format
```

The 400 is not retried, because the same request would fail the same way
three more times. The caller gets the error, the 662 tokens and the cost
already spent, and can report the problem to a person.

## Pinning the model

Each Claude model ID names one fixed version of the model. From the 4.6
generation on, an id such as `claude-sonnet-4-6` is itself that version,
and an updated model ships under a new id. Earlier models also have
aliases on the Claude API: `claude-sonnet-4-5` points to the newest dated
version with that number, such as `claude-sonnet-4-5-20250929`. The same
page notes that the serving systems around a model can change over time,
so small differences can appear even on a fixed id
(Model IDs and versioning, Claude Platform documentation). A fixed id doesn't last forever
either. Each id has its own retirement date, with at least 60 days'
notice for a public model, and after that date requests to it fail
(Model deprecations, Claude Platform documentation). Pinning lets you choose when the model
changes and plan the move. You still have to move before the retirement
date.

Choosing which model to pin is a trade between cost, speed and quality,
and a simple step can go to a smaller, cheaper model
(Claude Platform 101, Claude Academy) (From prototype to production: eval-gated deploys, rollout, cost, Agent Engineer Course). The fixture pins
`fake-helper-2-1`. It also has an alias, `fake-helper-latest`, which the
fake provider moved to `fake-helper-3-0` on 2026-06-01. The `alias` step
runs task `t1` on the alias one day before that date and on that date.

#### Example: The same code on two dates

Run this, and compare what you see with the output below.

```sh
python3 ops.py alias
```

Output:

```text
2026-05-31: fake-helper-2-1, calls 3, cost $0.004323
2026-06-01: fake-helper-3-0, calls 4, cost $0.006597
```

Nobody changed the code or the prompt. The newer model makes one more
call and writes a longer answer, and the run costs about half as much
again.

#### Pitfall: An alias in the configuration

A team writes an alias such as `fake-helper-latest` into its agent's
configuration, because it always gives them the newest model. On a date
the provider chose, the alias moves. The agent's answers get longer, it
makes an extra call on some tasks, and the cost per run jumps, with
nothing in the team's own history to explain it. Rule: pin an exact
model ID, and treat a new model as a change like a new prompt. It passes
the evaluation suite first, then goes to a small share of the traffic,
and reaches everyone only when its numbers hold.

## Exercise

Copy the folder and add a `table` step to `ops.py`. For each of the
five tasks in `TASKS` it calls `run` and prints one row with the task id,
the calls, the tokens and the cost. The last line is the average cost of
the five tasks. Find the task whose cost is far above the others, and
run it with `log=print` to see which call makes the difference. Name the
tool that causes it. A cost table per task type is the view that finds
this in production, where no one reads each run.

A good result: the other four tasks cost between $0.002364 and
$0.004323, one task costs $0.010635, and the average of the five is
$0.004804. So the costly task is more than twice the average you printed,
and about three times the average of the other four. Its trace shows a
small first call and a much larger second call, because the tool
returned a long result and the loop sent all of it to the model. Would a
token budget have stopped that run, and what budget would you set for
this agent?

Stretch: Change read_log so that it returns only the lines that report a stop or an error, and run the table again. How far does that task's cost drop, and what could the model no longer see?

## Recap

1. A loop sends its whole history again on every call, so the number of
   turns and the size of each tool result decide what a run costs. Keep a
   cost record per run from the usage the provider reports
   (Prompt caching, Claude Platform documentation) (From prototype to production: eval-gated deploys, rollout, cost, Agent Engineer Course).
2. A token budget stops a run with a named reason, and the caller still
   gets the tokens and the cost already spent. The check runs before a
   call, so a run can end a little over the budget.
3. Prompt caching writes the unchanging prefix once and reads it back at
   a fraction of the input price, but only while the prefix matches
   exactly (Prompt caching, Claude Platform documentation) (Pricing, Claude Platform documentation).
4. Retry rate limits, overload and timeouts with a wait, and use
   `retry-after` when the provider sends it. Report every other error to
   the caller (Claude API errors, Claude Platform documentation) (Rate limits, Claude Platform documentation).
5. Pin an exact model ID, and roll out a new model like any other change.
   An alias moves on the provider's date, and a pinned id retires on a
   date you are told about in advance (Model IDs and versioning, Claude Platform documentation)
   (Model deprecations, Claude Platform documentation).

#### Habit

Before an agent gets a new tool, look at the size of one real result from
it. The loop sends that result to the model again on every later call of
the run.

## References

- Prompt caching, Claude Platform documentation, https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- Pricing, Claude Platform documentation, https://platform.claude.com/docs/en/about-claude/pricing
- From prototype to production: eval-gated deploys, rollout, cost, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/11-from-prototype-to-production/
- Rate limits, Claude Platform documentation, https://platform.claude.com/docs/en/api/rate-limits
- Claude API errors, Claude Platform documentation, https://platform.claude.com/docs/en/api/errors
- Model IDs and versioning, Claude Platform documentation, https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions
- Model deprecations, Claude Platform documentation, https://platform.claude.com/docs/en/about-claude/model-deprecations
- Claude Platform 101, Claude Academy, https://academy.claude.com/courses/claude-platform-101