The demo is the easy part. An agent that plans, calls three tools, and returns a clean answer in front of stakeholders can be assembled in an afternoon. The version of that same agent that runs unattended for six months, against real data, inside a company that has an audit function, is a different piece of engineering.
MetaSys has shipped 76+ systems into production since 2019, and the agentic AI systems among them are where these notes come from. Below are the patterns that hold once real traffic arrives, and the ones that tend not to survive first contact with it.
What separates a demo agent from a production agent?
Constraints. A demo agent is judged on whether it can complete the task. A production agent is judged on what it does when it cannot: what it refuses, what it escalates, what it writes to the log, and how quickly a human finds out. Most of the engineering in a production agent lives in that second category, and almost none of it shows up in a demo.
The practical consequence is that the build order inverts. You do not build the agent and then add guardrails to it. You define the action surface, the escalation rules, and the evaluation criteria first. The agent is what fills the space you left for it.
Scope the tool surface before you tune the model
Every tool a system can reach should be declared, permissioned, and logged, and an agent should only be able to call the tools it was scoped to call. One of the most common architecture mistakes in early deployments is over-permissioning: handing an agent write access to a system it only ever needs to read.
Over-permissioning is attractive because it removes work: broad credentials granted up front mean nobody has to go back and add a scope later. They also mean the blast radius of one bad decision is the whole system rather than one record.
Read and write is the first cut. Per-action scoping is the second. An agent that can create a draft invoice does not need the ability to approve one, and an agent that can read a customer record does not need the ability to merge two of them. Each of those is a separate permission, and the agent should hold the smallest set that lets it finish the job.
Where do the human review gates go?
On the actions that are expensive to reverse, not on the ones the model finds difficult. Payments above a threshold, messages sent externally under the company name, deletions, and anything that creates a commitment to a third party all belong behind a gate, regardless of how confident the system is.
Teams frequently get this backwards and gate on model confidence instead. A stated confidence score is not a calibrated probability, and a system can be entirely certain about an action that is entirely wrong. Gate on consequence, then use confidence to decide what gets sampled for review inside the ungated set.
Calibration is the real work, and there is a measurement that tells you whether you got it right. Track what reviewers actually do at each gate. If they approve nearly everything without changing it, the gate is ceremony and it is costing you the latency it was meant to protect. If they reject a meaningful share, the agent is not ready for that action yet, and the gate is holding together a system that should still be in evaluation.
Evaluation is a launch gate, not a retrospective
Every system we ship carries an evaluation layer: automated eval sets, regression suites, and production dashboards, with accuracy, latency, and cost measured before launch and monitored after it. The ordering is the point. Evaluation that begins after deployment is not evaluation, it is incident response with better tooling.
The test set decides whether any of this works. A set built from clean examples will tell you the agent is excellent, and production will disagree within days. The useful set is assembled with the people who currently do the work by hand, because they are the ones who know which cases are hard, which are rare and expensive, and which look routine but are not.
Cost per unit of work belongs in the launch gate alongside accuracy. An agent that is accurate and unaffordable at volume is not a production system, it is a prototype with an invoice attached. That number is straightforward to measure in staging and painful to discover in the first full month of traffic.
Log the reasoning, not just the result
An audit trail that records what the agent did is not sufficient. The trail needs the inputs the agent saw, the tools it called and in what order, what each call returned, which branch it took, and where a human intervened. Without that, a wrong outcome is unexplainable, and an unexplainable outcome is unfixable.
Two things make this matter, and they arrive at different times. The first is debugging: when an agent produces a bad result three weeks after launch, the trace is the only artifact that tells you whether the model reasoned badly or a tool returned something unexpected. The second is compliance. For regulated systems our security engineering practice designs immutable, timestamped, queryable audit logs into the data model from day one, because retrofitting an audit trail after the auditor has already picked a date does not go well.
What does drift actually look like in production?
It looks like nothing. That is the whole problem. Agents rarely fail loudly. They get quietly worse while every dashboard stays green, because the system is still returning responses, still calling tools, and still completing runs. Nothing throws an error. The answers are simply less right than they were.
The usual causes sit outside your code. Input distribution shifts as the business adds product lines, new terminology, or an upstream vendor whose document format is slightly different. A model provider updates the model underneath you. A downstream API changes a field. None of these are code changes on your side, so none of them will be caught by anything that only runs in CI.
Continuous evaluation is the answer: sample production outputs, route a subset to human review, track accuracy, latency, and cost as time series rather than one-off measurements, and alert when a metric crosses an agreed threshold. You want to learn about degradation from a dashboard, not from a customer.
The patterns that do not hold up in production
- Full autonomy on day one. Deployments that launch with no gates tend to add them within weeks, at a worse moment and under more pressure than if they had been designed in.
- The mega-prompt. One enormous system prompt carrying every business rule the company has. It works until rule 41 contradicts rule 12, and by then nobody can tell which one the model followed or why.
- Confidence as a safety mechanism. A self-reported confidence score is a useful sampling signal and a poor control. When it is the only barrier between the agent and an irreversible action, the barrier is the model grading its own work.
- Evaluation deferred until after launch. Once the system is live, the only test set is production, and the cost of a failing case is a real customer rather than a red line in a report.
Where to start
One workflow. A bounded set of tools. A real evaluation set built with the people who do the work today. A review gate on the single action that would hurt most if it went wrong. Our own agent development sequence puts agent scoping in the first week, architecture design in weeks one to two, build and evaluation against real data in weeks two to six, and production deployment with observability and alerting from week six onward.
The scoping week is the one teams want to compress, and it is the one that determines whether the rest of the schedule holds. Everything expensive in an agentic system, the permission model, the gate placement, the evaluation criteria, is decided during that week, whether or not anyone decides it deliberately.
