How do you know when an agent that performed well in a sandbox is ready to touch live work? Many teams find out the hard way, usually in the same week the pilot's success deck meets the first real ticket queue.
A pilot answers one narrow question: can the model do the task on clean inputs, with a forgiving reviewer, against data nobody will miss? The first production phase answers a harder one — can the agent do the task every day, on messy inputs, against systems of record, with a named human accountable for what it writes?
A pilot proves the model can do the task on clean data. A first production phase proves the agent can do it daily against live systems of record, with measured exit criteria and a named human sign-off line.
That said, closing the gap between the two depends on scoping much more than on modeling. This post walks through the four decisions that close it: picking one workflow, writing exit criteria for the pilot, connecting to systems of record, and agreeing where human sign-off sits before the agent writes anything that matters.
Why A Sandbox Pilot Proves Less Than It Seems
The sandbox is designed to be forgiving, and that is its job. As we covered in our piece on agent sandboxing and isolated test environments, isolation lets you exercise tool calls and prompts without risking customer data or production state.
However, that same isolation hides the failure modes that dominate the first month of production. Some of the conditions a sandbox rarely reproduces include:
- Schema drift. The Salesforce or ServiceNow fixture you exported in June may no longer match the object model in October. A single custom-field rename is enough to break a tool call without raising an error.
- Stale-state reads. In production, a human and the agent can touch the same record within seconds of each other. The agent may act on a version of the case that is already out of date.
- Volume and concurrency. A pilot that processed 300 tickets over three weeks never saw a P99 latency spike from provisioned-throughput limits on Bedrock during a Monday-morning surge.
- Ambiguous inputs. Curated test sets lean toward the cases someone thought to write down. Live queues carry the forwarded email chain with three requests, two signatures, and an attachment.
All of these are reasons to read a strong pilot result as permission to scope a production phase — and nothing more. The production plan should be built around exactly these gaps.
A sandbox hides schema drift, stale-state reads, concurrency limits, and messy real-world inputs. Treat a strong pilot result as permission to scope production, and build the first phase around those four gaps.
How Do You Pick The First Production Workflow?
The first production workflow should be boring, bounded, and owned. Teams that launch three workflows at once tend to spend the first quarter unable to say which one caused the incident.
Keep in mind that the workflow that impressed in the demo is often a poor first choice, because demos reward breadth and production rewards containment. Instead, score candidate workflows against criteria such as:
- One system of record. The workflow reads from and writes to a single primary system — Zendesk, HubSpot, or one Snowflake schema — rather than orchestrating across four.
- Reversible output. Every action the agent takes can be undone with a compensating transaction or a simple revert. Draft replies, tags, and field updates with history all qualify.
- Measurable baseline. You already know how long humans take and how often they get it wrong. That gives the agent a real number to beat.
- A named owner. One operations lead owns the queue today and will own the agent's output tomorrow.
- Meaningful but tolerable volume. Several hundred items a week, for example, generates enough signal in a month without making a bad day catastrophic.
For instance, ticket triage and routing in Zendesk tends to score well on all five, while autonomous refund issuance across Stripe and Salesforce scores poorly on reversibility and system count. Our walkthrough of CRM agent automation covers which CRM workflows make strong first candidates.
Once the workflow is chosen, tie it back to the agent business case that justified the pilot. If the chosen workflow cannot move the metric the business case promised, the scope is wrong, and now is the cheapest time to find out.
Pick one workflow that touches a single system of record, produces reversible output, has a measured human baseline, and has one named owner. For phase one, containment matters more than impressiveness.
What Exit Criteria Should A Pilot Have To Clear?
Exit criteria are the written conditions under which the pilot ends and the production phase begins. Without them, pilots tend to drift into a permanent state of almost-ready, consuming budget and goodwill along the way.
Write them before the pilot starts, agree them with the workflow owner and whoever signs off on risk, and make each one a number. Here is a set we would expect to see ahead of a first production phase:
- Task accuracy against a labeled set. For example, 95% agreement with human reviewers on a held-out set of at least 200 real items. The scoring should be done by someone who did not write the prompts.
- Shadow-mode agreement. The agent runs on live inputs for two to four weeks, producing outputs humans can see but the system does not act on. Agreement should stay above threshold every week, not just on average.
- Latency and cost envelopes. P95 end-to-end latency and cost per completed task stay inside a stated budget at expected peak volume. A workable target might be under 8 seconds and under $0.40 per ticket.
- Failure handling verified. Every tool error, timeout, and malformed response routes to a defined path, as described in agent fallback design. None of them should end in a silent no-op.
- Model version pinned. The exact model version the criteria were measured on is pinned in configuration. A provider-side update should never change the agent's behavior between the go-live decision and go-live day.
Remember that these thresholds are yours to set, and the right numbers depend on how expensive an error is in the chosen workflow. What matters is that they exist in writing before anyone looks at the results, so the pilot cannot be graded on a curve.
Who grades the pilot. If the same engineer writes the prompts and grades the outputs, the accuracy number reflects their familiarity with the prompts as much as the agent's capability.
A blind reviewer who cannot see which output came from the agent removes that bias. At minimum, add a second reviewer on a random sample.
Exit criteria should be numeric and written before the pilot starts: accuracy on a held-out labeled set, weeks of shadow-mode agreement, P95 latency and cost-per-task envelopes, and verified failure handling.
Connecting To Systems Of Record Without Widening The Blast Radius
The sandbox usually ran against an export, a mock, or a read-only replica. Production means the agent holds credentials to the system the business treats as the truth — and that is where scoping discipline pays off.
Start with read access everywhere and write access in exactly one place. The agent should authenticate under its own service identity, scoped by an IAM role or a dedicated Salesforce integration user, so that every action is attributable — the pattern we lay out in agent identity and access management.
Next, put a thin, typed tool layer between the agent and the system of record. Instead of handing the model a general-purpose API client, expose narrow tools such as update_ticket_priority or add_internal_note, each with input validation, an idempotency key, and a retry policy.
Our guide to agent tool design covers how to size those tools. The integration work also surfaces questions the pilot never had to answer, including but not limited to:
- Write conflicts. When a human edits the record between the agent's read and its write, the agent needs a defined behavior. It should re-read, abort, or escalate — and that choice should be made in advance.
- Shared rate limits. Salesforce API limits and Zendesk rate limits are shared with every other integration on the org. An agent retry storm can starve unrelated systems that have nothing to do with the rollout.
- Audit trail. Every write needs a record of the input, the model version, the tool call, and the result, retained per your policy. Our post on agent audit trails details what to capture.
- Tenant and data boundaries. If the system holds data for multiple business units or clients, the agent's scope has to be enforced at the query layer. This is where tenant isolation in multi-tenant business integration actually holds, because a prompt instruction can be ignored and a scoped query cannot.
Overall, the goal is a connection where the worst thing the agent can do in a single action is small, logged, and reversible. If you cannot describe that worst case in one sentence, the write scope is still too wide.
Give the agent its own service identity, broad read access, and write access to one system through narrow typed tools with idempotency keys. Every write should be attributable, logged, and reversible.
Where Does The Human Sign-Off Line Go?
The sign-off line is the point in the workflow past which the agent's output takes effect without a human approving it. Agreeing where it sits — before the agent touches live work — is the decision that determines how much risk the first phase carries.
For a first production phase, we recommend placing the line close to the output. In practice, the agent drafts, classifies, or proposes, and a named human approves before anything reaches a customer or changes a financial or clinical record.
The line should then move on evidence, one step at a time. A useful progression for a first phase looks like this:
- Approve everything. A human reviews every agent action before it executes. Approval rates and edit distance are logged for every item.
- Approve by exception. Once agreement holds for an agreed period, low-risk categories execute automatically. High-risk categories still route to a reviewer.
- Sample and audit. For categories with a sustained clean record, a human reviews a sample after the fact. Automatic rollback stays available for anything the sample catches.
This mirrors supervised autonomy that expands one rung at a time on measured evidence, the pattern we documented for AI code review. It transfers cleanly to ticketing and CRM work because the underlying question — has this category earned less oversight? — is the same.
Be aware that the sign-off line also needs an owner and a demotion rule. If accuracy drops below the exit threshold for a week, the agent should drop a rung automatically, and the person who owns that call should be named in the runbook, as we describe in agent operator playbooks.
For phase one, place the sign-off line close to the output: the agent drafts or proposes and a named human approves. Move the line one rung at a time, only on measured evidence, with an automatic demotion rule.
Pilot Versus First Production Phase, Side By Side
The differences are easier to see in one place. Here is how the two stages typically compare across the dimensions an operations team will be asked about in review:
| Dimension | Sandbox pilot | First production phase |
|---|---|---|
| Data | Exports, fixtures, or replicas | Live system of record |
| Access | Broad, often shared credentials | Dedicated service identity with scoped write |
| Success measure | Demo quality and anecdotal review | Written exit criteria with numeric thresholds |
| Human role | Occasional spot check | Named approver at a defined sign-off line |
| Failure handling | Manual retry | Fallback paths, dead-letter queue, rollback |
| Observability | Logs in a notebook | Traces, metrics, and alerting |
| Cost tracking | Aggregate API bill | Cost per completed task against a budget |
Every row in the right-hand column is something you can build or agree before go-live. None of them depend on a better model.
What Should The First Thirty Days Of Production Look Like?
Once the agent goes live, the first thirty days are about confirming that the pilot's numbers hold under real conditions. Plan for a deliberately quiet launch, scheduled inside an agreed agent change window so the owning team is staffed and watching.
- Week one. Run at full human approval on a fraction of volume — 10% to 25% of the queue, routed by a feature flag. Compare agent outputs to human decisions daily.
- Week two. Expand to full volume if agreement holds. Review every escalation, timeout, and dead-letter entry with the workflow owner.
- Week three. Propose the first approve-by-exception categories, backed by logged agreement data. Get written sign-off from the risk owner before switching them on.
- Week four. Review cost per task, latency percentiles, and usage against the business case. The measures we describe in agent adoption metrics make that review concrete.
By the end of the month, you should have a production agent with a documented record and a defensible case for either expanding scope or holding steady. What's more, you will have built the release and monitoring muscle — covered in agent release management and agent observability — that every later workflow will reuse.
Use the first thirty days to confirm the pilot's numbers hold: start on 10–25% of volume at full approval, expand when agreement holds, then propose approve-by-exception categories with logged evidence.
Common Scoping Mistakes That Stall The First Phase
A handful of patterns show up repeatedly when a production phase stalls. Some of the most frequent include:
- Scoping by capability instead of workflow. Deploying an agent for customer support describes a capability. Triaging inbound Zendesk tickets into five queues describes a workflow you can actually measure.
- Moving the goalposts after the pilot. Exit criteria rewritten to fit the results carry no information. If a threshold was wrong, say so, reset it, and re-run shadow mode against the new number.
- Leaving cost for later. A per-task cost that looked trivial at pilot volume can become material at production volume. Our guide to AI agent cost governance covers budgets and alerts.
- No rollback rehearsal. If nobody has turned the agent off in staging and confirmed the queue drains back to humans cleanly, the kill switch is an assumption. Rehearse it before go-live, not during the first incident.
Each of these is avoidable with a one-page scope document signed before go-live. The document does not need to be long, but it does need to be specific.
For the broader operating model around that document, see our overview of AI agent operations and the three pillars of production AI. Both frame where the first phase fits in a longer rollout.
Frequently Asked Questions
How many workflows should the first production phase include?
One. A single workflow with one system of record lets you attribute every incident, cost spike, and accuracy change to a known cause. Add a second workflow only after the first has held its exit criteria for a full month.
How long should an agent run in shadow mode before going live?
Two to four weeks on live inputs is a reasonable range for most ticketing and CRM workflows. Volume decides it: you want enough real items, often several hundred, to see the long tail of messy cases.
Who should own the human sign-off line?
The operations lead who owns the workflow today, with written agreement from whoever carries risk for that system. Engineering builds the controls, but the business owner decides when the agent moves up or down a rung.
What happens if the pilot misses one of its exit criteria?
The pilot extends or the scope narrows, and the criteria stay put. Find whether the miss comes from inputs, tools, or the model, fix that layer, and re-run shadow mode before revisiting the go-live decision.
Should the agent get write access to the CRM on day one?
Only to the narrow fields the chosen workflow requires, through typed tools under a dedicated service identity. Broad read access is usually fine, while broad write access turns a small error into a data-cleanup project.
Scoping Your First Production Phase
Moving from pilot to production depends on four agreements: one workflow, numeric exit criteria, a contained connection to the system of record, and a sign-off line with an owner. Get those in writing, and the first production phase becomes a measured expansion of something you already understand.
If you're moving a sandbox pilot toward live systems of record and want a second set of eyes on the scope, the team at iSimplifyMe builds and operates production agent systems across CRM, ticketing, and data warehouse environments every week. Reach out for a working session — we'll pick the first workflow with you, write the exit criteria, and leave you with a sign-off plan your risk owner can approve.