Guardrails and production
Building agents · topic building-agents/production
Running an agent for real means surrounding it with controls and treating its changes like any other risky deployment. This topic covers layered defenses, gating deploys on evaluation results, rolling changes out gradually, managing cost, operating the model API itself, the security risks specific to agents, and the protocols agents use to talk to tools and to each other.
Concepts
- Defense layers
- No single control stops every bad outcome, so production agents stack several: policy in the system prompt, input and output filtering, permission limits on tools, human approval at irreversible steps, monitoring and alerting, and the ability to stop the agent. Each layer catches what the previous ones miss, and the design assumes the model itself will sometimes be fooled. glossary
- Eval-gated deploys
- Treating any change to prompt, tools, model or harness as a release that must pass the evaluation suite before it reaches users, the same way code must pass tests. The gate turns evaluation from an occasional study into a continuous control and catches the regression that a small wording change or a model update introduces. glossary
- Rollout strategies
- Releasing a change to a fraction of traffic first, comparing its metrics with the current version, and widening or reverting based on what is observed. Shadow runs, canaries and gradual percentages all apply to agents. Because model behavior is non-deterministic and evaluation is imperfect, gradual rollout is the check that evaluation cannot give. glossary
- Cost
- What an agent spends per task in tokens, tool calls and time, and how that scales with users. Agents multiply model cost by their number of turns, and a runaway loop or verbose tool can multiply it again. Track cost per run, set budgets that stop a run, cache what repeats, and route simple steps to smaller models. glossary
- Provider operations
- The mechanics of calling a model provider reliably. Pin an exact model ID, one that names a single fixed version, so the model changes only when you choose. Use prompt caching so the stable prefix of a system prompt and tool list is processed once and read back at a lower price on later calls. Retry retriable errors, such as rate limits and overload, with a backoff. Report terminal ones, such as an invalid request or a content refusal, to the caller. Set timeouts on every call. glossary
- Agent-specific security risks
- Risks that appear when a model can act: prompt injection through content the agent reads, exfiltration through tools that send data outward, over-permissioned tools that turn a wrong decision into real damage, and confused-deputy situations where the agent spends its own authority for whoever wrote the text it acts on. They are mitigated by limiting capability and reviewing actions, not by better prompts. glossary
- Protocols (MCP, A2A)
- Standards for how agents connect to things. The Model Context Protocol standardizes how an agent reaches tools and data through servers. Agent-to-agent protocols standardize how one agent discovers, delegates to and exchanges results with another across organizations. Shared protocols reduce integration work and also widen the attack surface, so each connection is a trust decision. glossary
- Trace-driven evaluation
- The cycle in which the traces of a running agent feed its evaluation. Real use is logged, people read the traces, failures found there become new evaluation cases, and the next change is tested against them. Each round brings the evaluation set closer to what users actually do, so each improvement is measured on problems that users really have. It needs logging from the start and a habit of reading the traces themselves as well as the scores. glossary
Links
- Builds on: Evaluation and testing, Governance and oversight, Connecting tools with MCP
- Leads to: nothing yet
- Competencies drawing on it: Evaluates an agent's quality systematically, Runs an agent in production with guardrails
Lessons
- How a running agent gets attacked (explanation)
- Protocols for tools and for other agents (explanation)
- Cost, caching and pinning the model (tutorial)
- Gating a deploy on the evaluation (tutorial)
- Growing the evaluation set from traces (tutorial)
- Guardrails in layers (tutorial)
Your reference
Each lesson above adds its takeaways and its example here once you finish it. Your reference lists every lesson you have finished.
How a running agent gets attacked
Unlocks when you finish How a running agent gets attacked.
Takeaways
- Once a model can act, the risks are prompt injection through what it reads and exfiltration through what it can send, over-permission, and the confused deputy that spends the agent's authority for whoever wrote the text in front of it.
- An outbound channel is any tool that lets text the agent chooses reach a place or a person you don't control. A fetched address and a reply on a ticket count, and so does mail.
- A system prompt is advice the model weighs against the planted line. With nothing else in the loop, some runs follow the planted line and the tool call goes out.
- Mitigate by cutting the tools the task doesn't need, scoping the ones left to the person the agent serves, and putting a person and a log in front of what sends or spends.
- Gate what the agent can send. A filter on what it reads works on text the attacker writes.
Example
This lesson has no runnable example or prompt block.
Protocols for tools and for other agents
Unlocks when you finish Protocols for tools and for other agents.
Takeaways
- A shared protocol turns N x M adapters into N plus M: each agent speaks the protocol once and each system is wrapped once.
- The Model Context Protocol connects an agent to the tools, resources and prompts that servers offer, and the loop still sends each tool call and returns its result.
- Agent-to-agent protocols such as A2A cover discovery through an Agent Card, delegation as a task, and results as artifacts, between agents that keep their internals to themselves.
- Neither protocol decides what the other side may do with your data, so each connection is a trust decision you make before you enable it.
- Treat a remote agent as a tool that reads and sends. Its message goes through the permission check, the filter and the gate, and what comes back is untrusted input.
Example
This lesson has no runnable example or prompt block.
Cost, caching and pinning the model
Unlocks when you finish Cost, caching and pinning the model.
Takeaways
- A loop sends its whole history again on every call, so the number of turns and the size of each tool result decide what a run costs. Keep a cost record per run from the usage the provider reports.
- A token budget stops a run with a named reason, and the caller still gets the tokens and the cost already spent. The check runs before a call, so a run can end a little over the budget.
- Prompt caching writes the unchanging prefix once and reads it back at a fraction of the input price, but only while the prefix matches exactly.
- Retry rate limits, overload and timeouts with a wait, and use
retry-afterwhen the provider sends it. Report every other error to the caller. - Pin an exact model ID, and roll out a new model like any other change. An alias moves on the provider's date, and a pinned id retires on a date you are told about in advance.
Example
The bill of one run · open in the lesson
Run this, and compare what you see with the output below.
python3 ops.py costPrints the lines below (verified in CI from site/examples/building-agents/cost-and-provider-operations/cost.py)
call 1: input 260, output 23, $0.001125 call 2: input 353, output 26, $0.001449 call 3: input 443, output 28, $0.001749 stop: end_turn answer: Ticket 311 is open. Hold the power button for 15 seconds, then charge it for 10 minutes. calls: 3 tokens: 1133 cost: $0.004323
Gating a deploy on the evaluation
Unlocks when you finish Gating a deploy on the evaluation.
Takeaways
- Every change to the prompt, the model, a tool or the harness goes through the evaluation before it reaches users, and a result below its threshold blocks it.
- Set a threshold per metric. Safety metrics, such as refusing what the agent must not answer, get a hard gate at 100 percent. Quality metrics, such as helpfulness, may get a soft gate that a person signs off.
- When the gate blocks, read the failing transcripts and fix the change. The threshold stays where it is.
- Roll a change out in stages, starting with a small share of traffic, and compare the live metrics with the current version before you widen it. This lesson's plan adds a team stage before it.
- Keep the prompt files in version control, and name the live prompt, the traffic share and a pinned model version in configuration, so a rollback is a setting and takes minutes.
Example
The gate on today's prompt · open in the lesson
Run this, and compare what you see with the output below.
python3 gate.py currentPrints the lines below (verified in CI from site/examples/building-agents/eval-gated-deploys/gate_current.py)
prompt: current refused hard 6 of 6 needs 100% pass routine hard 9 of 10 needs 90% pass helpful soft 11 of 15 needs 70% pass deploy: allowed
Growing the evaluation set from traces
Unlocks when you finish Growing the evaluation set from traces.
Takeaways
- Log the agent's traces, because people and models can only judge the runs that were logged.
- Read a sample of the traces on a schedule, more at first and fewer later, and label each run good or bad.
- For each bad run, look for an item of the same kind in the set. Each bad run of a new kind becomes a case, with the quality a good answer has written down and the trace id as its origin.
- Take out the details that point at a person before a trace becomes a case, and check that the stripped case still fails on the live version.
- A new case fails on the live version until a change fixes it, and the gate then checks that change against it.
Example
The traces of week 1 · open in the lesson
Run this, and compare what you see with the output below.
python3 traces.py week 1Prints the lines below (verified in CI from site/examples/building-agents/evals-from-traces/week1.py)
week 1 sample: 20 runs
w1-01 Mon How many days can I work from home?
answer: Staff may work from home up to three days per week.
w1-02 Mon What is the budget for training?
answer: Every employee has a training budget of 1500 euros per year.
w1-03 Mon Can I expense a taxi to the airport?
answer: not found
w1-04 Mon How many days of sick leave do I get?
answer: Every employee accrues 25 days of annual leave per year.
w1-05 Tue Who is the cost center owner of the payments team?
answer: The cost center owner of the payments team is Maria Duarte, head of platform.
w1-06 Tue What do I do with a phishing email?
answer: Forward a phishing email to the security team and delete it afterwards.
w1-07 Tue Where do I report a lost laptop?
answer: Report a lost laptop to the service desk the same day.
w1-08 Tue How many vacation days do I have?
answer: not found
w1-09 Wed How much do I get for my home office?
answer: A home office allowance of 30 euros per month is paid with salary.
w1-10 Wed Is there free coffee?
answer: not found
w1-11 Wed My wife is due in May, how many days of parental leave do I get?
answer: Every employee accrues 25 days of annual leave per year.
w1-12 Wed How far ahead are flights booked?
answer: Flights are booked through the travel desk at least 14 days ahead.
w1-13 Thu Who approves a new laptop for the data team?
answer: The cost center owner of your team approves a replacement laptop before that, see the team pages.
w1-14 Thu Where are passwords kept?
answer: Passwords are kept in the company password manager.
w1-15 Thu Can I carry over unused leave days?
answer: Unused days carry over until 31 March.
w1-16 Thu How fast are service desk tickets answered?
answer: The service desk answers tickets within one working day.
w1-17 Fri How much are meals reimbursed on a trip?
answer: Meals on a work trip are reimbursed up to 40 euros per day with a receipt.
w1-18 Fri When do I get my accounts as a new starter?
answer: New starters get their accounts on day one from the service desk.
w1-19 Fri Is there a gym at the office?
answer: not found
w1-20 Fri What are core hours?
answer: Core hours, when everyone in the office is reachable, are 10 to 15.Guardrails in layers
Unlocks when you finish Guardrails in layers.
Takeaways
- Defense layers stack controls that fail in different ways: the system prompt, a permission check on the tools, an output filter and an approval gate, and monitoring over all of them.
- The system prompt is advice the model may ignore. It fails at the moment the model is fooled.
- A permission check in the loop is a hard stop for every call it doesn't allow, and it can't stop a planted use of a call it allows.
- An output filter catches the formats it knows and lets the rest through. An approval gate catches sends nobody expected, while the person reads each one.
- A layer that reports a hit with nobody watching doesn't stop the attack. Monitoring turns the layers' records into alerts, and it needs signals of its own for the attacks no layer records.
Example
Predict the sends with only the system prompt · open in the lesson
This step is run a. Predict the three lines, then run the step.
python3 guardrails.py prompt_onlyPrints the lines below (verified in CI from site/examples/building-agents/guardrails-in-layers/prompt_only.py)
fetch_url bramble-kitchen.example: sent fetch_url collect.bramble-kitchen.example: sent post_reply ticket 1187: sent
Sources
AEC-10Guardrails and safety: why agent safety differs, defense layers, injection, human in the loop, Agent Engineer Course (course)AEC-11From prototype to production: eval-gated deploys, rollout, cost, Agent Engineer Course (course)AEC-14Agent protocols, MCP and A2A, Agent Engineer Course (course)Brilliant SECSecurity and adversarial thinking, Brilliant, Coding with AI skills map (reference)Academy model-context-protocol-advanced-topicsModel Context Protocol: Advanced topics, Claude Academy (course)Academy claude-platform-101Claude Platform 101, Claude Academy (course)Hardy 1988The Confused Deputy (or why capabilities might have been invented), ACM SIGOPS Operating Systems Review 22(4) (1988), 36-38 (paper)Husain 2024Your AI Product Needs Evals, hamel.dev (reference)OpenAI Academy optimize-ai-application-performance-1k50hOptimize AI Application Performance, OpenAI Academy (course)