Sit in on a production readiness review for an agent workflow and listen to what the room actually argues about. It is rarely model selection, and it is almost never the vector store.
The argument is over a question nobody has written down an answer to: how do we know this thing is working well enough to leave running over a long weekend? The traces are there, the token counts are there, the tool-call spans are all there — and none of them settle it.
That gap is the distance between observability and a service level objective. Your agent observability stack records what the agent did; an SLO states how often what it did has to be acceptable, and what happens when it is not.
An agent SLO is a target rate for acceptable outcomes — task completion, escalation, latency, and correction burden — measured over a rolling window. It turns a subjective judgment call into a number with a consequence.
Why Uptime SLOs Break On Non-Deterministic Systems
The SRE playbook most infrastructure teams already run assumes a binary, machine-checkable definition of success: the request returned a 200 inside the latency budget, or it did not. Availability and latency are cheap to measure precisely because correctness is treated as a property of the code, verified once at deploy time.
An agent breaks that assumption in a specific way. A workflow that reads a Zendesk ticket, queries Snowflake, and posts a refund of $4,300 instead of $430 will return a clean 200 in under two seconds, emit a perfectly formed trace, and burn no error budget at all under a conventional availability SLO.
What's more, the same input will not always produce the same output. Temperature, retrieval ordering, tool-call sequencing, and a model version bump you did not pin all move the result, which means a single passing test tells you far less than it did in a deterministic service.
Accordingly, agent SLOs have to be defined over distributions rather than individual requests. You are not asserting that this run will be correct; you are asserting that at least 94% of runs in the trailing 28 days met a written definition of correct, and committing to act when they do not.
The one-line test for an agent SLI: could the person who owns the budget for this workflow read the number and know whether to keep funding it? Availability passes that test for a database and fails it for an agent.
The Four Signals That Hold Up As Agent SLIs
The service level indicators that survive contact with a real agent workload are the ones a business owner recognizes without a tutorial. Four of them do most of the work, and they are chosen so that one cannot be gamed without visibly damaging another.
- Task completion rate. The percentage of runs that reached the intended end state — ticket resolved, record updated, report delivered — judged against a written rubric rather than an HTTP status. This is the headline SLI and the one most teams define at the wrong grain.
- Escalation rate. The percentage of runs the agent handed to a human on purpose, whether through an agent approval gate, a confidence threshold, or an explicit refusal. Escalation is a controlled outcome rather than a failure, which is exactly why it needs its own ceiling.
- Latency budget. Time-to-first-token for interactive surfaces and end-to-end wall-clock for the whole orchestration, tracked at P50, P95, and P99. A workflow with four tool calls and a retry policy will have a P99 an order of magnitude past its P50, and that tail is where your queue depth comes from.
- Correction burden. The share of completed runs a human subsequently edited, reversed, or redid, measured from the downstream system of record rather than the agent's own logs. This is the only one of the four your users are already tracking whether you ask them to or not.
Taken together, these four describe the shape of the workload rather than the health of the process serving it. Keep in mind that availability and error rate still belong on the dashboard — they simply stop being the thing that tells you whether the agent is doing its job.
Four SLIs cover most agent workloads: task completion rate, escalation rate, latency at P50/P95/P99, and correction burden. Track all four so a team cannot improve one by quietly wrecking another.
How Do You Measure Task Completion Without Reviewing Every Run?
The objection to a completion-rate SLO is always the same: someone has to decide what counts as complete, and that someone does not scale. The answer is a three-tier measurement ladder where each tier is cheaper and less precise than the one above it.
The bottom tier is deterministic assertions, run on 100% of executions and costing effectively nothing. Did the refund amount parse as currency and fall inside policy limits, did the Salesforce write return an ID, did the output validate against the JSON schema, did the run terminate without landing in the dead-letter queue.
The middle tier is a model-as-judge grader on a sample — typically 5% to 10% of runs, stratified so that low-volume, high-stakes paths are oversampled rather than lost in the average. At roughly $0.004 to $0.02 per graded run on a mid-tier model, a 200,000-run month costs somewhere between $80 and $400 to grade at a 10% sample, which is a rounding error against the workflow's own inference spend.
The top tier is human review, reserved for every escalation, every customer complaint, and a weekly replay of a pinned golden set of 100 to 300 cases. That golden set is the load-bearing piece: it is the only measurement that does not move when your traffic mix moves, and it is what connects your SLO to your agent evaluation harness.
Of course, the judge is itself a non-deterministic system and it drifts. Pin its model version, hold out a labeled set of roughly 50 cases where you already agree on the answer, and re-score that set every time you touch the judge prompt or the judge model.
Measure completion in three tiers: deterministic assertions on 100% of runs, an LLM judge on a 5-10% stratified sample, and human review of all escalations plus a weekly golden-set replay.
Correction Burden Is The Signal Your Users Already Track
Completion rate is what you measure; correction burden is what your users experience. A workflow can post a 96% completion rate against its own rubric while the support team quietly rewrites a third of its ticket responses before they go out.
That gap is the most useful diagnostic in the entire set, because it separates rubric error from agent error. When completion is high and correction burden is also high, the rubric is wrong — the agent is doing exactly what you told it to and you told it the wrong thing.
Instrument it from the downstream system of record, not from the orchestration layer. A DynamoDB stream or an EventBridge rule on the Zendesk, ServiceNow, or Salesforce object that captures edits within 24 hours of an agent write will give you a defensible number without asking anyone to fill in a form.
For instance, a reasonable starting target for an internal drafting workflow is that no more than 20% of completed runs are materially edited within a day, tightening to under 8% for anything that reaches a customer unreviewed. Be aware that a correction-burden number below 2% usually means nobody is looking rather than that the agent is perfect.
Correction burden is the share of completed agent runs a human later edits or reverses, measured from the system of record. High completion plus high correction means the rubric is wrong, not the model.
Latency Budgets For Workflows That Run In Minutes, Not Milliseconds
Agent latency behaves differently from request latency because the work is composed. A five-step workflow with two retrieval calls, a Bedrock invocation, a Salesforce write, and a retry policy has a wall-clock distribution built from five distributions, and the P99 is dominated by whichever step has the fattest tail.
Split the budget accordingly. Time-to-first-token governs whether an interactive surface feels alive and typically belongs under two seconds; end-to-end wall-clock governs queue depth and downstream commitments, and might reasonably sit at 45 seconds P95 for ticket triage or eight minutes for a nightly reconciliation run.
Set a hard timeout below the point where the work stops being useful, and make the timeout path an escalation rather than an error. A run that gives up at 90 seconds and hands a human a partial draft with its retrieved context attached is a much better outcome than one that burns four minutes and returns nothing.
Keep in mind that Lambda cold starts, throttling against provisioned throughput limits, and a model provider's own tail latency all land in your P99 without appearing anywhere in your code. Those belong in the budget as named contributors, not as unexplained variance.
Building An Error Budget When The Failure Mode Is A Wrong Answer
A conventional error budget is arithmetic: 99.9% availability over 30 days leaves about 43 minutes of downtime to spend. The agent equivalent works the same way, except the unit is an unacceptable outcome instead of a minute.
A workflow doing 20,000 runs a month against a 97% completion SLO has a budget of 600 unacceptable outcomes. However, treating all 600 as equivalent is the mistake that makes the whole exercise collapse, because a verbose summary and a $4,300 refund posted to the wrong account are not the same event.
The fix is a severity weighting applied before the budget math. Assign each failure class a multiplier so a single high-severity outcome consumes as much budget as dozens of low-severity ones, and the number starts to reflect actual exposure.
| Failure class | Example | Budget weight | Response |
|---|---|---|---|
| Cosmetic | Off-tone or verbose response, content correct | 1x | Log it, review in the weekly rubric pass |
| Incomplete | Agent stalls mid-workflow, item lands in the dead-letter queue | 3x | Retry with a compensating transaction, alert on rate |
| Wrong answer, reversible | Incorrect CRM field write, misrouted ticket | 10x | Page the on-call, roll back the write |
| Wrong answer, externally visible | Incorrect refund amount, bad commitment made to a customer | 50x | Freeze releases, open an incident |
| Policy or data violation | PII in an unapproved sink, out-of-policy tool call | Budget exhausted | Halt the workflow, escalate to security |
Under this scheme, three externally visible wrong answers in a month consume 150 units of a 600-unit budget, and one policy violation ends the quarter's freedom to ship regardless of how clean the rest of the numbers look. Accordingly, the weights are a business decision that belongs to the workflow owner rather than a tuning knob for the platform team.
Build the budget in unacceptable outcomes, not minutes: 20,000 runs at a 97% SLO allows 600. Weight each failure by severity so one wrong customer-facing write costs 50x a cosmetic miss.
What Should Actually Happen When The Budget Burns?
An error budget with no attached consequence is a chart. The policy is the product, and it has to be written down before the first breach, while nobody has a release they are trying to protect.
A workable escalation ladder runs in four steps:
- 50% burned. Notify the workflow owner and require a written cause note in the weekly review. Release cadence is unchanged.
- 75% burned. New prompts, tools, and model-version bumps ship to shadow mode only, running against live traffic without taking action, per your agent release management policy.
- 100% burned. Feature freeze on the workflow, mandatory rollback to the last model version and prompt revision that held the SLO, and reduced autonomy through tighter confidence thresholds.
- Severity-1 event. Halt the workflow and run it through agent incident response, preserving the trace, the tool calls, and the retrieved context for the post-incident review.
Note that the 100% response deliberately trades throughput for correctness rather than pausing the workflow outright. Raising the escalation rate is almost always a better lever than a hard stop, because it keeps the workflow producing while humans absorb the uncertainty.
Write the burn policy before the first breach: 50% burned triggers a cause note, 75% ships changes to shadow mode only, 100% freezes releases, rolls back the model pin, and raises escalation.
Where Agent SLOs Quietly Go Wrong
Most failed SLO programs fail at definition rather than at instrumentation. Four failure modes account for nearly all of it.
- Completion defined at the wrong grain. Measuring per tool call instead of per business task produces a 99.4% number on a workflow that resolves half its tickets, because 180 successful API calls can still add up to one unhappy customer.
- Escalation used as an outlet. When escalation carries no cost, the cheapest way to hit a completion target is to escalate anything hard, which turns a green dashboard into a growing queue for a human team.
- Judge drift. An unpinned grader model or a quietly edited judge prompt will move your completion rate several points with no change to the agent at all, and that movement is indistinguishable from a real regression.
- Survivorship in the sample. Grading only the runs that finished excludes the timeouts, the dead-letter queue, and the sessions the user abandoned, which are exactly the runs where the failure lives.
All four share a root cause: an SLI that is easier for the platform team to produce than for the business owner to recognize. The test for any proposed agent SLI is whether the person who owns the workflow would accept it as evidence in a budget conversation.
A Thirty-Day Path To Your First Agent SLO
You do not need a mature platform to start, and you should not set targets before you have baseline data. Here is a sequence that works on a single workflow:
- Week one — write the rubric. Three to five sentences defining what a complete run looks like for this specific workflow, agreed with the business owner and stored in version control next to the prompt.
- Week two — instrument without targets. Ship the deterministic assertions, the judge sample, and the correction-burden capture from the system of record, then publish the numbers with no thresholds attached. Resist every request to explain a bad day.
- Week three — read the baseline. You will typically find completion 5 to 15 points below what the team assumed, plus a correction burden nobody had counted, and that is your real starting position.
- Week four — set the target and the policy. Pick an objective slightly above the baseline rather than at the aspiration, define the severity weights, write the burn policy, and name the person allowed to declare the budget spent.
From there the objective ratchets on a review cadence rather than continuously — quarterly is common, and moving it mid-window destroys the comparison you are trying to build. What's more, the same rubric and severity table that drive the SLO feed directly into agent cost governance, because unacceptable outcomes and wasted inference spend usually turn out to be the same runs.
Frequently Asked Questions
What is a realistic task completion SLO for a production agent?
Start at 90-95% for internal read-only workflows and 97-99% for customer-facing writes, measured on a graded rubric rather than pass/fail. Set the first target from four weeks of baseline data, not from ambition.
How do you measure agent correctness at scale?
Use a three-tier ladder: deterministic assertions on every run, an LLM judge on a 5-10% stratified sample, and human review of all escalations plus a weekly replay of 100-300 pinned golden cases.
What is an error budget for a non-deterministic agent?
It is the allowed volume of unacceptable outcomes in a window — 3% of 20,000 monthly runs equals 600. Weight each incident by severity so one wrong customer-facing write costs far more budget than one vague summary.
Should escalation to a human count as an agent failure?
No. Escalation is a controlled outcome with its own target, usually 5-15%. Only silent wrong answers burn the correctness budget — otherwise the cheapest way to hit completion is to escalate everything.
How long should an agent SLO measurement window be?
Use a rolling 28-day window for the objective and a 7-day window for alerting. Shorter windows make sampling noise look like a regression; longer windows can hide a bad model-version rollout for weeks.
Set The Number Before The Incident Sets It For You
If you are standing up your first agent SLO and want a second set of eyes on the rubric before the targets get political, the team at iSimplifyMe builds and operates production agent systems across CRM, ticketing, and data warehouse environments every week, and publishes the rest of this material under AI agent operations.
Reach out for a working session — we will define your completion rubric, weight your failure classes, and leave you with an instrumented error budget and a written burn policy.