Skip to content
AI Training

Caching and effort

A team uses an assistant all day to answer questions about one long contract. Two settings change what that day costs, and neither changes the questions they ask. The first sets how much work the model puts into each answer. The second lets the long start of every request, the instructions and the contract, be paid for at a small part of the normal price. In this lesson we choose an effort level for a kind of work and find out what makes the cache stop working. You install and run nothing: every count on the page is in a table.

The lesson on reading the token counts showed that each request in a conversation sends the whole history again. This lesson starts from there.

What the effort level changes

Section titled “What the effort level changes”

The effort level is a setting that tells the model how many tokens to spend on its response [1]. On one vendor’s models the levels run from low through medium and high to max, and some models add one more level between high and max. If nobody sets it, most of those models run at high and one recent model at medium [1].

The setting applies to everything the model writes, and not only to its thinking. The vendor’s page lists the text of the answer, the requests for tools and the thinking, when the model thinks, as all affected by it [1]. The page expects fewer tool calls and shorter reports at a lower level. At a higher level it expects more tool calls, a plan explained before the model acts and a more detailed summary of what it did [1].

All of that is output, and output is the expensive kind of token. For an agent there is a second cost. Every extra round of tool calls is one more request, and each request sends the whole history again. Higher effort adds input tokens that way. Higher effort costs more tokens and more time, and lower effort is faster and cheaper [1].

The setting guides the model and does not set a hard limit. A hard problem still gets some thinking at a low level, but it gets less of it [1]. And on at least one model, the vendor says, a lower level changes how much it thinks but does not reliably make the visible answer shorter [1]. When you want a short answer, ask for one in the prompt.

Checkpoint · choice

What does a lower level change?

Section titled “What does a lower level change?”

An agent renames the files in a folder to one pattern. You lower its effort level from high to low. What changes in the agent’s work?

Choosing a level

Section titled “Choosing a level”

The vendor’s advice follows the kind of work. A low level suits simple, clear tasks where speed and cost matter most, such as a helper agent that does one small job. High suits hard reasoning, difficult problems and agent work. Max is for the tasks that need the deepest thinking, when the number of tokens is no concern, and medium sits between high and low [1].

That is a starting point, and the same page says the right level depends on the task, so try it on your own work before you decide [1]. Run a few typical requests at two levels and compare the results. If the lower level gives the same quality, keep it. Choose the level for a kind of work, such as “drafting emails” or “reviewing contracts”, and not for each question. The section on the cache below shows why the level should then stay the same for the rest of the session.

Checkpoint · match

Which level would you start with?

Section titled “Which level would you start with?”

Match each task to the effort level you would try first.

What the cache saves

Section titled “What the cache saves”

Look again at the day with the contract. The requests start with the same instructions and the same contract, and the end of each is new. The vendor processes that same start again for every question, and charges for it each time. Prompt caching keeps the processed start of a request for a short time. When a later request begins with exactly the same tokens, the model reuses the stored work for that part instead of processing it again [2].

The tool that sends the request decides whether to cache. Software that sends requests to one vendor’s models turns caching on in the request [3], and one coding agent turns it on by itself [4]. The lesson on reading the token counts mentioned the usage reports that then split the input count by what the cache did with each token.

The two kinds of cached input have their own prices. On that vendor’s price list, reading from the cache costs a tenth of the normal input price on most models and less on a few. Writing to the cache costs a quarter more than normal input, for a cache that lasts five minutes [3] [2]. So the first request pays a little extra to store the start, and each request after it that reads the start pays much less.

Here is the contract day, priced twice. The counts are round numbers written for this lesson. The input and output prices are the example prices for a large model from the lesson on what every token costs, $6 per million input tokens and $30 per million output tokens. The cache prices follow the vendor’s rule above, and the first line of the table lists all four. The table assumes that the tool caches everything up to the newest message. Each request then reads what the request before it sent and writes the rest. The instructions are 1,000 tokens.

one chat, priced without and with a prompt cache
prices per million tokens: input $6.00, output $30.00, cache write $7.50, cache read $0.60
request                                      input written    read output  no cache    cached
1. pastes a contract, asks what it covers   21,000  21,000       0    500   $0.1410   $0.1725
2. asks when the contract ends              21,515     515  21,000    120   $0.1327   $0.0201
3. asks about the penalties                 21,655     140  21,515    300   $0.1389   $0.0230
4. asks for a summary for the manager       21,980     325  21,655    400   $0.1439   $0.0274
5. after lunch, asks who signed it          22,395  22,395       0     80   $0.1368   $0.1704
total                                      108,545  44,375  64,170  1,400   $0.6933   $0.4133

The totals are computed from the summed counts before rounding, so a column added up by hand can differ from them by $0.0001.

Read the written and read columns first. Request 1 writes all 21,000 tokens to the cache, so it costs more with the cache ($0.1725) than without it ($0.1410). Request 2 reads those 21,000 tokens back and writes only the first answer and the new question, and it costs $0.0201 instead of $0.1327. Requests 3 and 4 work the same way.

Now read the output column. It is the same with and without the cache. Caching changes nothing about what the model writes. The answer comes out exactly as it would without the cache [2], and every output token is billed at the normal output price. The new text at the end of each request is never read from the cache either, because no earlier request held it.

What breaks the cache

Section titled “What breaks the cache”

Request 5 came after a lunch break, and it read nothing. A cache entry lasts five minutes by default, and each read starts the five minutes again at no extra cost. A cache that lasts an hour is also available, at a higher write price [2]. One coding agent’s cost guide gives the same five minutes by default when the agent is billed by the token. On a subscription it gives an hour, which drops to five minutes once the subscription draws on usage credits. It also says that the first message after a longer break processes the whole history again [4]. So request 5 wrote all 22,395 tokens again and cost more than it would have without a cache.

A break is one way to lose the cache, and a change to the start of the request is the other. The start of a request is built in a fixed order: the tool descriptions first, then the instructions, then the messages. The cache matches only a start that is identical up to the point it stored [2]. A change at any point ends the match there, and everything after it is processed and paid for again.

  • Changing the tools that the assistant can use ends the match at the very start, so nothing is read from the cache [2].
  • Editing the instructions ends it after the tool descriptions, so the whole conversation is processed again.
  • Changing the effort level between requests ends the match at least before the messages [2], because the level is written into the prompt the model reads [1]. Some newer models can change the level partway through a conversation and keep the cache, with a setting that is still in beta. The new level then holds until another change. On other models the vendor’s advice is to choose a level at the start of a session and keep it there [1].
  • Adding a text message at the end keeps the match. Everything before it is the same, so it is read from the cache. An image is an exception: adding one anywhere ends the match before the messages [2].

Keep the effort level the same within a session for that reason. Choose the level for the kind of work when the session starts, and when you need another level for different work, start that work in its own session.

A very short start isn’t cached at all. Each model has a minimum length, from about five hundred to a few thousand tokens [2]. The cache saves the most in a long session that repeats a large start, and makes little difference to a short chat.

Checkpoint · multi-choice

What did the cache do in this chat?

Section titled “What did the cache do in this chat?”

In the table above, which two statements are true?

Select exactly 2.

Checkpoint · sort

Will the next request read the cache?

Section titled “Will the next request read the cache?”

Checkpoint · scenario

A hard question in the middle of the session

Section titled “A hard question in the middle of the session”

You have drafted emails at low effort all morning in one session. Now you have one hard question, unrelated to the emails, that needs high effort. What do you do?

Exercise

On paper or in a note, take these three tasks: turning a list of meeting dates into a table, planning the move of a twelve-person team to a new office over three months, and checking whether a 40-page tender meets each of 25 requirements. For each one, choose an effort level and say in one line what that level changes in the model’s work. Then picture one afternoon in a single chat about the tender, which you pasted at the start. Write which part of each request prompt caching makes cheaper, which part it leaves at the normal price, and two things that would make the next request pay for the whole tender again. Ten minutes is enough.

A good result chooses low for the table and high for the plan and the tender, and says that a higher level means more thinking and more tool calls, at more tokens and time. It adds that you would try the level on a few real cases first. For the cache, it names the instructions, the tender and the earlier messages as the part read from the cache. The answers and the thinking stay at the normal output price. Each new question and the answer before it are never read from the cache in that request: they are written to it, at a little more than the normal input price. It names two of these as what breaks the cache: a break longer than the cache lifetime, an edit to the instructions, a change of tools, or a change of effort level. To close, ask yourself: in your own work, which long start do you send again most often?

Recap

  1. The effort level tells the model how many tokens to spend, and it applies to all of its output: the thinking, the tool calls and the answer [1].
  2. Start low for simple, clear work and high for hard or open work, and test the level on a few real cases before you decide [1].
  3. Prompt caching makes the unchanged start of a request cheap to read again. The output is billed as before, and the new text at the end is never read from the cache. A five-minute cache stores that new text at a quarter more than normal input [2] [3].
  4. The cache matches from the start and ends at the first change. A changed tool or instruction, an effort level changed for the whole request, or a break longer than the cache lifetime makes the next request process the history again [2] [4].
  5. Choose the effort level for a kind of work when a session starts, and keep it until the session ends.

You can now

  • Chooses an effort level for a kind of work and keeps it fixed
  • Explains what prompt caching saves and what breaks it

References

Section titled “References”
  1. Anthropic. Effort. Claude Platform documentation. Reference. Claude docs effort
  2. Anthropic. Prompt caching. Claude Platform documentation. Reference. Claude docs prompt-caching
  3. Anthropic. Pricing. Claude Platform documentation. Reference. Claude docs pricing
  4. Anthropic. Manage costs effectively. Claude Code documentation. Reference. Claude Code costs