Sit in on the third week of an approved agent pilot and watch where the calendar time actually goes. It is not model selection, and it is not the retrieval layer.
It is a senior operator in a conference room being asked to explain, out loud, what they do when the nightly Snowflake load fails at 2:14 a.m. Halfway through the explanation they stop, because they have just realized they have never written any of it down.
That meeting is runbook capture, and almost nobody budgets for it. It is the single most common reason a pilot approved in March is still not in production in September.
The agent business case was built on a workflow diagram with six boxes in it. The real workflow has fourteen steps, four decision branches, three approval thresholds that vary by customer tier, and one step that exists only because a vendor API returns HTTP 200 on a write that silently failed.
None of that is in the diagram. All of it is in the head of the person who has carried the pager for that system for four years.
Runbook capture is the process of converting undocumented operator knowledge into a parameterized, testable procedure an agent can execute. It specifies inputs, tool calls, decision branches, failure handling, and rollback for every step.
Why Tribal Knowledge Blocks Production Agents
An agent cannot execute a procedure that does not exist in a form it can read, and it cannot be evaluated against a standard nobody has written. Those are the same problem wearing two hats.
When a workflow lives only in an operator's head, the organization has no ground truth. There is no artifact to diff agent behavior against, so every review becomes a judgment call by the one person who already knows the system.
That person becomes the bottleneck for the pilot that was supposed to free them. Across the CRM, ticketing, and data warehouse workflows we build, capture consumes roughly 50 to 70 percent of elapsed time from approval to first production run, while the orchestration build itself consumes 15 to 25 percent.
This is why capture belongs in the AI change management plan rather than the engineering backlog. It is an organizational extraction problem wearing an engineering costume, and staffing it like a sprint task is how it slips two quarters.
Most agent pilots stall in capture, not in modeling. The workflow exists only in an operator's head, so there is no ground truth for the agent to be evaluated against and no artifact for review.
What Runbook Capture Actually Produces
Capture produces a structured, parameterized, testable artifact — one an orchestrator can bind tools to and an evaluation harness can score. A prettier wiki template does not get you there.
Every captured procedure carries the same five components:
- Typed inputs. Every value the procedure consumes is named, typed, and sourced. Not "the account," but tenant_id from the Salesforce record, environment from the deployment manifest, and retry_budget as an integer with a default.
- Atomic steps bound to a tool call. One step equals one observable action against one system. If a step maps to two API calls in two services, it is two steps, because failure between them needs its own handling.
- Decision branches with explicit predicates. The operator's "if it looks stuck" becomes a predicate: queue depth above 500 messages for more than 10 minutes, or SQS approximate age of oldest message above 900 seconds.
- Failure handling and compensating actions. Each step declares what happens on timeout, on partial success, and on a poisoned payload — including the compensating transaction that unwinds a half-applied change.
- A success test. The procedure ends with an assertion a machine can evaluate, not a human impression that things are back to normal.
All of these together are what make a procedure executable rather than merely readable. Miss any one of them and you have a document that an agent can summarize but cannot run.
Where Your Existing Documentation Falls Short
Most infrastructure teams already have runbooks, and they are genuinely useful — to humans who share the missing context. The gap shows up the moment you hand the same page to an orchestrator with a tool registry and no institutional memory.
| Dimension | Wiki runbook | Agent-executable procedure |
|---|---|---|
| Inputs | Implied by prose | Named, typed, and sourced |
| Step granularity | "Restart the connector" | One step per tool call, per system |
| Branches | "If it looks stuck" | Explicit predicate on a measurable signal |
| Failure path | "Escalate to the team" | Retry policy, dead-letter queue, compensating action |
| Reruns | Assumed safe | Idempotency key per mutating step |
| Success test | Operator judgment | Machine-evaluable assertion |
| Versioning | Page history | Pinned version bound to a release |
The point of the comparison is not that the wiki page is wrong. It is that the wiki page was written for a reader who could fill the gaps, and capture is the work of removing every gap that required a human to fill it.
How Do You Extract A Procedure From An Operator Who Has Never Written It Down?
Asking someone to describe their own expertise from memory produces the happy path and almost nothing else. Four techniques, run in sequence, produce a draft that survives contact with production:
- Incident archaeology. Pull twelve months of PagerDuty incidents and ServiceNow tickets for the target system and read the resolution notes. The exception branches nobody remembers in a conference room are sitting in the timestamps of the 3 a.m. threads.
- Narrated execution. Sit with the operator during a real run and have them talk through every check, including the ones they do not think count. The tab they glance at before approving a change is a precondition, and it belongs in the procedure.
- Transcript mining. Slack channel history and on-call handoff notes carry the informal conditionals — the customer tiers that get different treatment, the vendor whose retries need a longer backoff, the region that always fails first.
- Adversarial interview. Once a draft exists, hand it to a second operator and ask them where it breaks. This pass reliably surfaces two to five branches the first operator considered too obvious to mention.
Run in that order, the four techniques compound: archaeology gives you the failure taxonomy, narration gives you the preconditions, transcripts give you the exceptions, and the adversarial pass gives you the branches. Skipping any one of them shifts that discovery into shadow mode, where it costs more.
Extract procedures through incident archaeology, narrated execution, transcript mining, and an adversarial second-operator review. Interviews alone produce the happy path and miss the exception branches entirely.
Why The First Draft Of Every Runbook Is Wrong
The first draft captures what the operator does when the system fails in the way it usually fails. Production has a longer tail than that, and the tail is where agents cause damage.
Three categories go missing with near-perfect consistency. Unlogged preconditions come first — the operator checks whether a migration is in flight, or whether the vendor status page is green, and has never once written that check down because it takes two seconds.
Approval thresholds come second. A refund under $500 is automatic, a refund over $5,000 needs a director, and the middle band depends on customer tier — which is exactly the logic that has to become explicit before you can wire agent approval gates around it.
Escalation semantics come third. "Escalate to the platform team" hides a decision about severity, a paging policy, and a handoff payload, all of which have to be specified before the agent can participate in agent incident response without making the incident worse.
Plan for two to three revision cycles per procedure and the schedule holds. Plan for one and the pilot slips, because the discovery happens anyway — just later, in front of an audience.
Parameterization Is Where Capture Efforts Stall
Teams usually get through the narrative capture and then stop, because parameterization is the part that feels like software engineering. It is, and it is also the part that determines whether the procedure runs once or a thousand times.
Take a single line from a real runbook: restart the connector. Parameterized, it becomes a step that names the connector by ID, takes the environment as an input, checks in-flight message count before acting, carries an idempotency key so a retried invocation does not double-restart, declares a 90-second timeout, and asserts a post-condition on consumer lag.
That expansion is not overhead. It is the contract the agent tool design layer binds to, and it is what lets the same procedure run against staging and production without a fork.
The step smell test: if a step says "check whether it looks right," "clean it up," or "escalate if needed," it is not yet a step. Each of those phrases is a decision an operator makes from context the agent does not have.
Parameterization also exposes coupling nobody had named. When two procedures both mutate the same DynamoDB item, capture is where you discover you need a compensating transaction and a defined ordering — long before an agent finds out for you at P99.
A procedure is agent-executable when every step names its inputs, its tool call, its success test, and its failure branch. If a step says "check whether it looks right," it is not yet a step.
What Does Runbook Capture Cost?
The honest answer is more than the pilot budget assumed, and less than a second failed pilot. Here are the ranges we plan against, measured in senior operator hours rather than engineering hours, because operator time is the constrained resource:
| Workflow tier | Steps and branches | Senior operator hours | Calendar time |
|---|---|---|---|
| Simple | 5-10 steps, 1-2 branches | 16-24 | 2 weeks |
| Standard | 10-20 steps, 3-6 branches | 40-60 | 4-6 weeks |
| Complex | 20+ steps, cross-system, approvals | 80-140 | 8-12 weeks |
At a loaded rate of $150 to $250 per hour for the people who actually hold the knowledge, a standard workflow lands between $6,000 and $15,000 of capture cost before a single agent invocation. Add the facilitation and engineering time and a realistic all-in figure for one standard workflow is $18,000 to $35,000.
Compare that against what the same workflow burns unattended. Teams that skip capture and go straight to build typically spend that money anyway, in rework, and add six to ten weeks of calendar time doing it.
Budget 40 to 60 senior operator hours per standard workflow for first-pass capture, plus two to three revision cycles. All-in, that is roughly $18,000 to $35,000 before any agent runs in production.
How Do You Know A Runbook Is Ready To Hand To An Agent?
Readiness is measured, not declared. Three gates decide it, and all three are cheap relative to a bad production run.
The first gate is a golden-case suite. Take fifteen to thirty historical incidents, replay them against the captured procedure on paper, and confirm the branch predicates route each one the way the operator actually routed it — the same discipline that governs agent evaluation more broadly.
The second gate is shadow mode. The agent proposes the next action, the operator executes, and you record agreement rate; two weeks at 90 percent or better across the natural incident mix is a defensible bar.
The third gate is failure rehearsal. Deliberately break a dependency, confirm the procedure lands the run in the dead-letter queue with a readable reason, and confirm your agent observability surfaces it inside your alerting SLA rather than at the next standup.
Clear all three and the handoff is a scheduling decision rather than a leap of faith. Clear two and you are shipping the third gate's discovery into production.
A runbook is ready when it survives shadow mode: the agent proposes, the operator executes, and their choices agree on 90 percent or more of runs across two weeks of real traffic.
Who Owns The Runbook After Capture?
Captured procedures decay faster than most teams expect. An API version bump, a revised approval threshold, a vendor migration, or a change to the on-call rotation can each invalidate a branch that has been quietly executing for months.
Ownership therefore sits with the team that owns the underlying system, not with the AI team that facilitated the capture. This is a standing line item in the agent operating model, alongside model-version pinning and access review.
Bind the procedure version to the release, the way you would any other deployable artifact — the same discipline described in agent release management. A runbook that can change without a release is a runbook whose behavior your agent audit trails cannot reconstruct six months later.
Where a procedure crosses team boundaries, name the owner of each segment and the payload that moves between them. That boundary contract is the same one described in agent handoff patterns, and undefined ownership at a boundary is where stale-state reads come from.
Ownership belongs to the team that owns the underlying system, not the AI team. Runbooks drift the moment an API version, approval threshold, or on-call rotation changes without a matching revalidation.
A Capture Sequence That Holds Up
The sequence below is what we run on a standard workflow, and the ordering matters more than the tooling. Here is the order:
- Pick one high-volume, low-blast-radius workflow. Resist the executive favorite if it touches billing or PHI on the first pass.
- Run incident archaeology before the first interview, so you walk in with the failure taxonomy already drafted.
- Narrate one real execution end to end, recording every check the operator makes without being asked.
- Draft the procedure with typed inputs, atomic steps, explicit predicates, and failure branches on the same day, while the run is fresh.
- Hand the draft to a second operator for the adversarial pass and expect to add branches.
- Build the golden-case suite from historical incidents, then run two weeks of shadow mode.
- Assign the owner, pin the version to a release, and schedule revalidation against change events rather than the calendar.
Seven steps, four to six weeks, one workflow in production with an artifact you can point at during an audit. The second workflow moves roughly 40 percent faster, because the branch taxonomy and the tool contracts carry over.
Runbook Capture Questions, Answered
Is runbook capture the same as writing documentation?
No. Documentation explains a procedure to a human; capture specifies it for a machine, with typed inputs, explicit branch predicates, a success test, and a failure path for every step.
How many workflows should we capture before the first production run?
One. Capture a single high-volume, low-blast-radius workflow end to end, ship it, then use what you learned about branch density and drift to estimate the next five.
Can an agent write its own runbook from logs and transcripts?
It can draft one, and that is a useful accelerant. It cannot supply the preconditions operators never log, so narrated execution and an adversarial review still gate the draft.
What is the most common missing piece in a captured runbook?
Failure branches. Teams capture the happy path in the first session, then discover on the second pass that the operator has six exception behaviors, four of which were never ticketed.
Who should own runbook capture inside the organization?
The team that owns the underlying system, with the AI team facilitating. Ownership anywhere else guarantees drift the moment an API version, approval threshold, or on-call rotation changes.
How often do captured runbooks need to be revalidated?
Tie revalidation to change events rather than the calendar. Any upstream API version bump, schema change, threshold revision, or vendor migration should trigger a shadow-mode rerun.
Bring Us The Workflow Nobody Has Written Down
If you have an approved agent pilot that has been stuck at 80 percent for a quarter, the blocker is almost certainly capture rather than architecture. The team at iSimplifyMe runs capture sessions and builds production AI agent operations across CRM, ticketing, and data warehouse environments every week.
Reach out for a working session and we will do three things: narrate one real execution with your operator, return a parameterized draft procedure with its branch predicates and failure paths written out, and give you an hours estimate for the remaining workflows in the queue.