Tokens, cost and effort
Concepts · topic concepts/tokens-and-cost
Every request to a model is counted and billed in tokens. This topic covers the counts a request reports, why each request in a long session costs more than the one before, how prompt caching makes an unchanged start of a request cheap, and how an effort setting trades thoroughness against tokens.
Concepts
- Input and output tokens
- The two counts every response reports. Input tokens are everything the request sent: instructions, tool descriptions, the history and the new message. Output tokens are what the model wrote, including tool requests and any thinking, and each one costs several times as much as an input token. With prompt caching, the input count is split further into tokens written to the cache, tokens read from it and the rest. Reading these counts shows where the cost of a request comes from. glossary
- Growing session cost
- Because the model is stateless, each request in a session sends the whole conversation again, and an agent sends a new request after every batch of tool results. So each request carries everything before it, and a question late in a long session costs more than the same question in a new one. Starting fresh for an unrelated task resets the history, and compacting shrinks it at the price of one large summary request. glossary
- Prompt caching
- Reusing the processing of the opening part of a request when a later request starts with exactly the same tokens. The reused part is billed at a small fraction of the normal input price. The opening part has a fixed order: tool descriptions, then the system prompt, then the messages. After a change, everything from that point on is cached again, at more than the normal input price. An unused cache expires after five minutes, or after one hour at a higher write price. glossary
- Effort level
- A setting that tells the model how many tokens to spend on a response, from low to max. Higher effort gives more thinking and more thorough tool use, and costs more tokens and time. Lower effort is faster and cheaper and suits simple work. The setting guides the model but does not set a hard limit, and on at least one model it does not reliably change the length of the answer. Changing it in one conversation can invalidate the prompt cache. glossary
Links
- Builds on: How language models work
- Leads to: nothing yet
- Related: Capabilities and limits, Choosing models and tools
- Competencies drawing on it: Reasons about what a model request costs and why
Lessons
- Caching and effort (explanation)
- Reading the token counts (tutorial)
Your reference
Each lesson above adds its takeaways and its example here once you finish it. Your reference lists every lesson you have finished.
Caching and effort
Unlocks when you finish Caching and effort.
Takeaways
- The effort level tells the model how many tokens to spend, and it applies to all of its output: the thinking, the tool calls and the answer.
- Start low for simple, clear work and high for hard or open work, and test the level on a few real cases before you decide.
- Prompt caching makes the unchanged start of a request cheap to read again. The output is billed as before, and the new text at the end is never read from the cache. A five-minute cache stores that new text at a quarter more than normal input.
- The cache matches from the start and ends at the first change. A changed tool or instruction, an effort level changed for the whole request, or a break longer than the cache lifetime makes the next request process the history again.
- Choose the effort level for a kind of work when a session starts, and keep it until the session ends.
Example
This lesson has no runnable example or prompt block.
Reading the token counts
Unlocks when you finish Reading the token counts.
Takeaways
- Every request reports two counts. Input tokens are everything the request sent: instructions, tool descriptions, the history and the new message. Output tokens are everything the model wrote, thinking and tool requests included.
- An output token costs several times an input token, so read the two counts separately and price each one.
- Each request in a conversation holds the whole history, so the input count only goes up. A short question late in a long chat can cost more than a long one at the start.
- An agent sends a request for every step of its loop, and every tool result goes into the input of the requests after it.
- Start a new conversation for an unrelated task. Compacting makes later requests smaller, and the summary is one large request of its own.
Example
This lesson has no runnable example or prompt block.
Sources
Claude docs prompt-cachingPrompt caching, Claude Platform documentation (reference)Claude Code costsManage costs effectively, Claude Code documentation (reference)Claude docs effortEffort, Claude Platform documentation (reference)Claude docs messagesMessages, Claude Platform documentation (reference)Claude docs pricingPricing, Claude Platform documentation (reference)OpenAI Academy optimize-ai-application-performance-1k50hOptimize AI Application Performance, OpenAI Academy (course)