# Caching and effort

> See what an effort level changes in everything the model writes, choose a level for a kind of work, and read a long chat priced with and without a prompt cache to find what the cache saves and what makes it stop working.

Page: https://schubergphilis.github.io/ai-training/concepts/caching-and-effort/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

A team uses an assistant all day to answer questions about one long
contract. Two settings change what that day costs, and neither changes the
questions they ask. The first sets how much work the model puts into each
answer. The second lets the long start of every request, the instructions
and the contract, be paid for at a small part of the normal price. In this
lesson we choose an effort level for a kind of work and find out what makes
the cache stop working. You install and run nothing: every count on the
page is in a table.

The [lesson on reading the token counts](https://schubergphilis.github.io/ai-training/concepts/reading-token-counts/#a-short-chat-request-by-request)
showed that each request in a conversation sends the whole history again.
This lesson starts from there.

## What the effort level changes

The **effort level** is a setting that tells the model how many tokens to
spend on its response (Effort, Claude Platform documentation). On one vendor's models the
levels run from low through medium and high to max, and some models add
one more level between high and max. If nobody sets it, most of those
models run at high and one recent model at medium (Effort, Claude Platform documentation).

The setting applies to everything the model writes, and not only to its
thinking. The vendor's page lists the text of the answer, the requests for
tools and the thinking, when the model thinks, as all affected by it
(Effort, Claude Platform documentation). The page expects fewer tool calls and shorter
reports at a lower level. At a higher level it expects more tool calls, a
plan explained before the model acts and a more detailed summary of what
it did (Effort, Claude Platform documentation).

All of that is output, and output is the expensive kind of token. For an
agent there is a second cost. Every extra round of tool calls is one more
request, and each request sends the whole history again. Higher effort adds
input tokens that way. Higher effort costs more tokens and more time, and
lower effort is faster and cheaper (Effort, Claude Platform documentation).

The setting guides the model and does not set a hard limit. A hard problem still
gets some thinking at a low level, but it gets less of it
(Effort, Claude Platform documentation). And on at least one model, the vendor says, a lower
level changes how much it thinks but does not reliably make the visible
answer shorter (Effort, Claude Platform documentation). When you want a short answer, ask for
one in the prompt.

#### Checkpoint: What does a lower level change?

An agent renames the files in a folder to one pattern. You lower its effort
level from high to low. What changes in the agent's work?

- Only the thinking gets shorter, and the tool calls and messages stay as they were.
- The agent reads fewer input tokens, because it skips part of its instructions.
- The agent writes fewer tokens: less thinking, fewer tool calls and terser messages around them.
- The agent stops at a fixed number of tokens, even in the middle of a hard step.

## Choosing a level

The vendor's advice follows the kind of work. A low level suits simple,
clear tasks where speed and cost matter most, such as a helper agent that
does one small job. High suits hard reasoning, difficult problems and agent
work. Max is for the tasks that need the deepest thinking, when the number
of tokens is no concern, and medium sits between high and low
(Effort, Claude Platform documentation).

That is a starting point, and the same page says the right level depends on
the task, so try it on your own work before you decide (Effort, Claude Platform documentation).
Run a few typical requests at two levels and compare the results. If the
lower level gives the same quality, keep it. Choose the level for a kind of
work, such as "drafting emails" or "reviewing contracts", and not for each
question. The section on the cache below shows why the level should then stay
the same for the rest of the session.

#### Checkpoint: Which level would you start with?

Match each task to the effort level you would try first.

Statements:

- Turn a list of 200 addresses into one fixed format
- Compare two supplier contracts and list every place they conflict
- Sort incoming emails into three named folders, as a helper agent
- Find out why a monthly report total is wrong, across several spreadsheets

Options:

- Low effort
- High effort

## What the cache saves

Look again at the day with the contract. The requests start with the
same instructions and the same contract, and the end of each is new. The
vendor processes that same start again for every question, and charges for
it each time. **Prompt caching** keeps the processed start of a request for
a short time. When a later request begins with exactly the same tokens,
the model reuses the stored work for that part instead of processing it
again (Prompt caching, Claude Platform documentation).

The tool that sends the request decides whether to cache. Software that
sends requests to one vendor's models turns caching on in the request
(Pricing, Claude Platform documentation), and one coding agent turns it on by itself
(Manage costs effectively, Claude Code documentation). The [lesson on reading the token
counts](https://schubergphilis.github.io/ai-training/concepts/reading-token-counts/#two-counts-on-every-request)
mentioned the usage reports that then split the input count by what the
cache did with each token.

The two kinds of cached input have their own prices. On that vendor's price
list, reading from the cache costs a tenth of the normal input price on most
models and less on a few. Writing to the cache costs a quarter more than
normal input, for a cache that lasts five minutes (Pricing, Claude Platform documentation)
(Prompt caching, Claude Platform documentation). So the first request pays a little extra to
store the start, and each request after it that reads the start pays much
less.

Here is the contract day, priced twice. The counts are round numbers
written for this lesson. The input and output prices are the example prices
for a large model from the [lesson on what every token
costs](https://schubergphilis.github.io/ai-training/concepts/what-tokens-cost/#what-a-token-costs), $6 per million input
tokens and $30 per million output tokens. The cache prices follow the
vendor's rule above, and the first line of the table lists all four. The
table assumes that the tool caches everything up to the newest message.
Each request then reads what the request before it sent and writes the
rest. The instructions are 1,000 tokens.

```text
one chat, priced without and with a prompt cache
prices per million tokens: input $6.00, output $30.00, cache write $7.50, cache read $0.60
request                                      input written    read output  no cache    cached
1. pastes a contract, asks what it covers   21,000  21,000       0    500   $0.1410   $0.1725
2. asks when the contract ends              21,515     515  21,000    120   $0.1327   $0.0201
3. asks about the penalties                 21,655     140  21,515    300   $0.1389   $0.0230
4. asks for a summary for the manager       21,980     325  21,655    400   $0.1439   $0.0274
5. after lunch, asks who signed it          22,395  22,395       0     80   $0.1368   $0.1704
total                                      108,545  44,375  64,170  1,400   $0.6933   $0.4133
```

The totals are computed from the summed counts before rounding, so a
column added up by hand can differ from them by $0.0001.

Read the written and read columns first. Request 1 writes all 21,000
tokens to the cache, so it costs more with the cache ($0.1725) than without
it ($0.1410). Request 2 reads those 21,000 tokens back and writes only the
first answer and the new question, and it costs $0.0201 instead of $0.1327.
Requests 3 and 4 work the same way.

Now read the output column. It is the same with and without the cache.
Caching changes nothing about what the model writes. The answer comes
out exactly as it would without the cache (Prompt caching, Claude Platform documentation),
and every output token is billed at the normal output price. The new text
at the end of each request is never read from the cache either, because no
earlier request held it.

## What breaks the cache

Request 5 came after a lunch break, and it read nothing. A cache entry
lasts five minutes by default, and each read starts the five minutes again
at no extra cost. A cache that lasts an hour is also available, at a higher
write price (Prompt caching, Claude Platform documentation). One coding agent's cost guide
gives the same five minutes by default when the agent is billed by the
token. On a subscription it gives an hour, which drops to five minutes
once the subscription draws on usage credits. It also says that the
first message after a longer break processes the whole history again
(Manage costs effectively, Claude Code documentation). So request 5 wrote all 22,395 tokens again and cost
more than it would have without a cache.

A break is one way to lose the cache, and a change to the start of the
request is the other. The start of a request is built in a fixed order: the
tool descriptions first, then the instructions, then the messages. The
cache matches only a start that is identical up to the point it stored
(Prompt caching, Claude Platform documentation). A change at any point ends the match there,
and everything after it is processed and paid for again.

- **Changing the tools** that the assistant can use ends the match at the
  very start, so nothing is read from the cache (Prompt caching, Claude Platform documentation).
- **Editing the instructions** ends it after the tool descriptions, so the
  whole conversation is processed again.
- **Changing the effort level** between requests ends the match at least
  before the messages (Prompt caching, Claude Platform documentation), because the level is
  written into the prompt the model reads (Effort, Claude Platform documentation). Some newer
  models can change the level partway through a conversation and keep the
  cache, with a setting that is still in beta. The new level then holds
  until another change. On other models the vendor's advice is to choose a
  level at the start of a session and keep it there (Effort, Claude Platform documentation).
- **Adding a text message at the end** keeps the match. Everything before
  it is the same, so it is read from the cache. An image is an exception:
  adding one anywhere ends the match before the messages
  (Prompt caching, Claude Platform documentation).

Keep the effort level the same within a session for that reason. Choose the
level for the kind of work when the session starts, and when you need
another level for different work, start that work in its own session.

A very short start isn't cached at all. Each model has a minimum length,
from about five hundred to a few thousand tokens (Prompt caching, Claude Platform documentation).
The cache saves the most in a long session that repeats a large start, and
makes little difference to a short chat.

#### Checkpoint: What did the cache do in this chat?

In the table above, which two statements are true?

Select exactly 2.

- Request 1 cost less with the cache, because the contract was stored in it.
- Request 2 cost less with the cache, because the contract was read from it.
- Request 5 cost less with the cache, because the contract had been stored that morning.
- The output tokens of every request cost the same with and without the cache.

#### Checkpoint: Will the next request read the cache?

Buckets:

- The next request reads the history from the cache
- The next request processes the history again

Items:

- You ask a new question two minutes after the last answer
- You paste one more paragraph of text at the end of the conversation
- You edit a rule in the instructions at the top
- You come back after a two-hour meeting
- You switch the effort level from high to low for the next question
- You turn on a tool the assistant did not have before

#### Checkpoint: A hard question in the middle of the session

You have drafted emails at low effort all morning in one session. Now you
have one hard question, unrelated to the emails, that needs high effort.
What do you do?

- Switch this session to high for the question, then back to low.
- Ask the hard question in this session at low effort.
- Ask the hard question at high effort in a new session.
- Switch this session to max, so every later question is answered well.

#### Pitfall: Polishing the instructions during the session

A person pastes a long report into an assistant and asks questions about it
all afternoon. Every few questions she opens the assistant's instructions and
improves one sentence. The usage report shows that each request after an
edit wrote the whole report to the cache again. The instructions come
before the conversation, so every edit ended the cached match there. The
rule: finish the instructions and choose the effort level before a long
session, and put a change in a new message at the end.

## Exercise

On paper or in a note, take these three tasks: turning a list of meeting
dates into a table, planning the move of a twelve-person team to a new
office over three months, and checking whether a 40-page tender meets each
of 25 requirements. For each one, choose an effort level and say in one
line what that level changes in the model's work. Then picture one
afternoon in a single chat about the tender, which you pasted at the start.
Write which part of each request prompt caching makes cheaper, which part
it leaves at the normal price, and two things that would make the next
request pay for the whole tender again. Ten minutes is enough.

A good result chooses low for the table and high for the plan and the
tender, and says that a higher level means more thinking and more tool
calls, at more tokens and time. It adds that you would try the level on a
few real cases first. For the cache, it names the instructions, the tender
and the earlier messages as the part read from the cache. The answers and
the thinking stay at the normal output price. Each new question and the
answer before it are never read from the cache in that request: they are
written to it, at a little more than the normal input price. It names two
of these as what breaks the cache: a break longer than the cache lifetime,
an edit to the instructions, a change of tools, or a change of effort
level. To close, ask yourself: in your own work, which long start do you
send again most often?

## Recap

1. The effort level tells the model how many tokens to spend, and it
   applies to all of its output: the thinking, the tool calls and the
   answer (Effort, Claude Platform documentation).
2. Start low for simple, clear work and high for hard or open work, and
   test the level on a few real cases before you decide (Effort, Claude Platform documentation).
3. Prompt caching makes the unchanged start of a request cheap to read
   again. The output is billed as before, and the new text at the end is
   never read from the cache. A five-minute cache stores that new text at a
   quarter more than normal input
   (Prompt caching, Claude Platform documentation) (Pricing, Claude Platform documentation).
4. The cache matches from the start and ends at the first change. A changed
   tool or instruction, an effort level changed for the whole request, or a
   break longer than the cache lifetime makes the next request process the
   history again (Prompt caching, Claude Platform documentation) (Manage costs effectively, Claude Code documentation).
5. Choose the effort level for a kind of work when a session starts, and
   keep it until the session ends.

#### Habit

The next time you start a long session with an assistant or an agent, choose its effort level for the kind of work first, and leave it there until the session ends.

## References

- Effort, Claude Platform documentation, https://platform.claude.com/docs/en/build-with-claude/effort
- Prompt caching, Claude Platform documentation, https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- Pricing, Claude Platform documentation, https://platform.claude.com/docs/en/about-claude/pricing
- Manage costs effectively, Claude Code documentation, https://code.claude.com/docs/en/costs