Reasons about what a model request costs and why

Concepts · competency concepts/reasons-about-cost

Taught in: the Concepts course

Draws on: Tokens, cost and effort, What an agent is

Learning objectives

Claim	Why	Example
The learner tells the input tokens from the output tokens in a usage report and knows that an output token costs more.	The two kinds are priced differently, so the same total can mean very different bills [1].	A summary job reports 40,000 input and 500 output tokens, and the learner sees that the long document, and not the short summary, is most of the cost.
The learner finds the part of the input that was read from the cache.	Input read from the cache is billed at a small fraction of the normal price, so the input count alone overstates the cost [2].	A report shows 30,000 tokens read from the cache and 800 new input tokens, and the learner explains that most of the request was the same as the one before.
The learner looks at the counts before guessing why a task was expensive.	The counts show which part of the request grew, and a guess usually blames the wrong part.	The learner expected the long answers to be the problem, and the counts show the cost is a large document attached to every message.

Claim	Why	Example
The learner explains that the tool sends the whole conversation with every request, so each request carries all the text before it.	The model is stateless, so a long history is paid for again on every request [3].	Asked why a one-line question late in a long chat used so many tokens, the learner says the question was sent together with an hour of history.
The learner starts a new conversation for an unrelated task.	The old history adds to the cost of every new request and does not help with the new task.	After finishing the report, the learner opens a new chat for the unrelated email instead of continuing in the long one.
The learner counts the requests an agent sends, which is more than the prompts they typed.	An agent sends a new request after each tool result, and each of those carries the history [4].	The learner sees one instruction lead to a dozen tool calls and expects about a dozen requests in the usage report.

Claim	Why	Example
The learner picks a low effort level for simple, well-defined work and a higher one for hard or open problems.	Higher effort spends more tokens on thinking and on tool use, and simple work gains nothing from that [5].	Reformatting a list of addresses runs at low effort, and comparing the terms of two contracts runs at high effort.
The learner tries a level on a few real examples of the work before settling on it.	The right level depends on the task, and the vendor's own advice is to test on your own work.	The learner runs three typical requests at medium and at high effort, sees no difference in quality, and keeps medium.
The learner keeps the level fixed within one conversation and changes it only between kinds of work.	Changing the effort level between requests can invalidate the prompt cache, so the next request writes the whole history to the cache again, at more than the normal input price [5] [1].	The learner sets the level when starting the review session and leaves it there, and uses another level in a separate drafting session.

Claim	Why	Example
The learner explains that a request reuses the cache only when it starts with exactly the same tokens as a recent request.	The cache stores the opening part of a request, and any difference ends the match at that point [2].	The learner explains why two requests with the same long document but a different first instruction both write the document to the cache instead of reading it.
The learner puts content that stays the same at the start of a request and content that changes at the end.	A change invalidates the cache from that point on, so changing text near the start wastes everything after it.	The learner puts the shared reference document before the question, so the next question reuses the document from the cache.
The learner names what breaks the cache: changed instructions or tools, a changed setting such as the effort level, and a pause longer than the cache lifetime.	Each of these makes the next request pay again for everything from the change on [3].	After a long lunch break, the first message in the old session costs much more than the one before the break, and the learner knows why.

Anthropic. Pricing. Claude Platform documentation. Reference. Claude docs pricing
Anthropic. Prompt caching. Claude Platform documentation. Reference. Claude docs prompt-caching
Anthropic. Manage costs effectively. Claude Code documentation. Reference. Claude Code costs
Anthropic. How the agent loop works. Claude Code documentation. Reference. Claude Code agent loop
Anthropic. Effort. Claude Platform documentation. Reference. Claude docs effort