Take a fifteen-step agent workflow — ticket triage, lead enrichment, claims intake, the shape matters less than the step count — running 40,000 executions a month in production. Eleven of those steps are deterministic in everything but implementation: parse a payload, classify an intent into one of six buckets, normalize a date, confirm a required field is present, format the response.
All fifteen call the same model endpoint, because that is how the workflow was wired during the pilot and nobody has had a reason to unwire it. The four steps that genuinely need frontier reasoning are subsidizing eleven that would return identical output from a model priced at a fraction of the rate.
Run the arithmetic against your own invoice rather than an industry average: take your blended cost per execution and multiply it by the share of steps doing string manipulation instead of reasoning. In the stacks we audit, that share usually lands between 60% and 80%, and the per-call spread between a frontier model and a small one sits near an order of magnitude.
Cost is only the visible half. A single hard-wired endpoint also means every step of every workflow shares one throttling ceiling, one provider incident, one region, and one model-version deprecation notice.
What A Model Routing Layer Actually Is
The routing layer is an indirection between what the workflow declares and what actually gets called. It is the difference between a step that names a model and a step that names a job.
Multi-model routing is an indirection layer that maps each workflow step to a logical model role, then resolves that role to a concrete model, version, and region at call time. Retiering a step becomes a config change.
The closest analogue in infrastructure you already run is service discovery. Your workflow does not hard-code an IP for the payments service; it references a name, and a resolver binds that name to a host at call time under a policy you own.
Model routing applies the same pattern to a logical role. The step declares that it needs an intent classifier, and the registry resolves that role to a pinned model version, a provider, a region, a max-token ceiling, and a timeout at the moment of the call.
Keep in mind that this is an architectural change with almost no workflow surface area. If your agent orchestration layer already separates step definitions from tool bindings, you are most of the way there.
Why Single-Endpoint Stacks Get Expensive And Brittle
Hard-wiring an endpoint is the correct decision during a proof of concept and a liability once the workflow carries volume. The failure modes it produces are predictable, and they arrive in a consistent order.
The three that show up in production reviews most often include but are not limited to:
- Reasoning-priced string manipulation. Steps that classify, extract, or reformat get billed at the same rate as steps that plan across constraints. This is the largest single line item most teams find when they instrument their agent cost governance for the first time.
- One blast radius for the whole graph. Every step shares one rate limit, one provisioned-throughput allocation, and one provider status page, so a 429 storm on a classification step takes down the reasoning steps queued behind it.
- A capability ceiling that applies everywhere. Whatever your chosen model is weakest at — long-context recall, strict JSON adherence, a specific language, tool-call precision — becomes a property of the entire workflow rather than a property of one step.
All of these share a root cause: the deployment made one decision on behalf of every step, and every step now inherits the consequences. Routing addresses all three through the same mechanism, which is turning a deployment-wide decision into a per-step one.
Capability Tiering: Matching Steps To Model Classes
Before you can route, you have to classify. Capability tiering is the exercise of grouping steps by what they actually demand rather than by what the pilot happened to give them.
Capability tiering sorts agent steps by reasoning demand — deterministic, extractive, generative, and open-ended reasoning — then assigns each class a model tier. Most steps land below the frontier tier.
A four-tier split covers nearly every production workflow we have taken through this exercise. The table below shows what belongs in each tier and what you should be watching once it is there.
| Tier | Step types | Model class | Primary metric |
|---|---|---|---|
| Tier 0 | Regex matching, date parsing, enum mapping, schema validation, arithmetic | No model — plain code | Deflection rate |
| Tier 1 | Intent classification, entity extraction, short summarization, routing decisions | Small fast model (Haiku-class, Llama-class, Titan-class) | Accuracy against golden set, P95 latency |
| Tier 2 | Customer-facing drafting, multi-field synthesis, moderate tool selection | Mid-tier model (Sonnet-class, mid-tier GPT) | Human-review pass rate |
| Tier 3 | Multi-constraint planning, ambiguous tool selection, exception handling, code generation | Frontier reasoning model (Opus-class, frontier GPT) | Escalation rate, cost per completed workflow |
Note that Tier 0 is where the largest savings usually sit, and it is the tier teams skip. What's more, a step that a model gets almost right and a regex gets exactly right is not a routing problem at all.
The first routing win is usually no model at all. Before you retier a step to a cheaper endpoint, check whether it needs inference in the first place.
Enum mapping, date normalization, required-field checks, and unit conversion are deterministic code wearing a prompt.
Once Tier 0 is drained, the remaining classification is straightforward. Steps that select from a closed set go to Tier 1, steps that produce prose a human will read go to Tier 2, and steps where the correct action is genuinely unknown before the model runs go to Tier 3.
How Per-Step Selection Works Without Touching The Workflow
The mechanism is a model registry plus a resolution policy, and it needs three pieces to be operationally safe. Each of them is boring on its own, which is the point.
- A role-keyed registry. Every step references a logical role — intent-classifier, ticket-summarizer, escalation-planner — and the registry holds the binding from that role to a provider, model ID, version pin, region, and inference parameters.
- A resolution policy evaluated at call time. The policy can key on tenant, environment, current cost burn against budget, latency SLO, or an incident flag, which is what lets you shed a tier during a provider outage without shipping a deploy.
- Model-version pinning held in config, not code. Roles pin to an explicit model version so a provider's point upgrade cannot quietly change your output distribution, and so promoting a new version arrives as a reviewable diff.
The workflow definition never names a model under this arrangement, which is the entire reason a retier does not require a rewrite. Changing a step from Tier 3 to Tier 1 is an edit to one registry entry, reviewed and shipped through the same agent release management path as any other config change.
On AWS, the registry usually resolves to Bedrock model IDs with the inference profile and region attached, which keeps the routing decision inside the same VPC, IAM, and CloudTrail boundary you already audit. Teams running this pattern on Bedrock specifically should read the Bedrock agent patterns breakdown for how a registry interacts with action groups.
Fallback Chains And What They Should Never Do
Fallback and escalation get conflated constantly, and conflating them is how routing layers produce silent quality regressions. They solve different problems and need separate instrumentation.
Fallback handles availability: a 429, 5xx, or timeout reroutes the same request to an equivalent-tier model. Escalation handles difficulty, promoting a step to a stronger tier. Meter them separately.
A fallback chain should move laterally rather than downward. If your Tier 1 classifier is throttled, the fallback target is another Tier 1 model at a different provider or region, not an unvalidated cheaper model that happens to have capacity.
Escalation moves upward and should be explicit in the step definition: a confidence threshold, a schema-validation failure, or a refusal triggers a single retry at the next tier. Be aware that escalation is a cost multiplier, so an escalation rate climbing past roughly 15% on a Tier 1 step usually means the step was tiered wrong rather than that the model is underperforming.
The failure mode to design against is the silent downgrade. A fallback that quietly serves a weaker model during an incident and returns output that passes schema validation while being substantively wrong is the hardest class of defect to catch after the fact, which is why the model and version actually used belong in your agent audit trail on every step rather than only the failed ones.
Accordingly, steps that write — creating a record, sending a message, moving money, updating a clinical or financial system — should fail closed instead of falling back. Route them to a dead-letter queue with the idempotency key intact and let your agent incident response runbook make the call, rather than letting a routing policy decide at 3 a.m.
What Routing Actually Breaks
Routing is not free, and its costs land in places single-model teams have never had to instrument. Every one of these is manageable, but each is a real workstream rather than a footnote.
Routing adds four workstreams: prompt portability, structured-output differences between providers, per-role evaluation sets, and cost accounting in dollars rather than tokens. Instrument before you route.
- Prompt portability. A system prompt tuned against one model's instruction-following will not transfer cleanly, and small models typically need shorter instructions, more explicit output contracts, and fewer nested conditionals.
- Structured-output semantics. Tool-call formats, JSON modes, and refusal behavior differ per provider, so a step that reliably returns valid structured output on one model can produce schema drift on another under the same prompt.
- Evaluation surface. Your validation suite now has to cover role-by-model combinations instead of one model, which means per-role golden sets and a regression gate on every pin change — the subject of our agent evaluation guide.
- Cost accounting. Tokenizers differ enough between providers that token counts stop being comparable, so per-token dashboards lose their meaning the moment you route.
- Latency distribution. Cheaper is not automatically faster, and some small models carry worse time-to-first-token or higher cold-start variance than the frontier model they replaced, which moves your P95 in ways a cost dashboard will never surface.
All of this adds up to a single prerequisite: you cannot route what you cannot measure. Teams that add routing before they add per-step agent observability end up unable to explain their own cost or quality curves.
How Do You Know Your Routing Is Working?
The metric most teams reach for — total model spend — is the one that will mislead you first, because it moves for reasons unrelated to routing quality. A routing layer needs its own instrument panel.
Track cost per completed workflow, per-role escalation rate, fallback invocation rate, per-role quality against a golden set, and P95 end-to-end latency. Token counts are not comparable across providers.
Cost per completed workflow is the only cost figure that survives routing, because it absorbs retries, escalations, and tier changes into one denominator you can compare month over month. Escalation rate per role tells you whether your tiering was correct, and fallback invocation rate tells you whether your provider mix is actually buying the availability you paid for.
Quality needs a per-role golden set — 50 to 200 labeled examples per role is usually enough to detect a meaningful regression — scored on every pin change and on a schedule against sampled live traffic. Remember that a model swap can hold accuracy steady while shifting output format in ways that break downstream parsing, so score format conformance separately from correctness.
A Rollout Sequence That Does Not Require A Rewrite
The sequencing matters more than the tooling, because each step below de-risks the one after it. Here is the order we run it in:
Adopt routing in this order: instrument per step, tier every step, swap model references for logical roles with no behavior change, build golden sets, shadow the candidate, then cut over one role. The workflow definition changes once.
- Instrument per step, not per workflow. Capture model, version, token counts, latency, and outcome for every step before changing anything, and let it run for at least one full traffic cycle.
- Classify every step into a tier. Mark Tier 0 candidates first, because the steps that need no inference at all are the cheapest and safest wins available to you.
- Introduce the indirection with zero behavior change. Replace hard-coded model references with logical roles and point every role at the incumbent model, so the diff ships without altering a single output.
- Build the golden sets. Sample real traffic per role, label it, and establish the incumbent's baseline score before any candidate model touches the workflow.
- Shadow the candidate. Run the cheaper model on live traffic without serving its output, and compare it against the incumbent on the same requests rather than on a static public benchmark.
- Cut over one role with escalation enabled. Pick the highest-volume Tier 1 candidate, enable escalation to the incumbent tier on validation failure, and watch escalation rate for a full cycle before touching a second role.
- Expand, then add availability fallback. Only once two or three roles are stable should you add cross-provider fallback, because a fallback path you have never exercised is an outage you have not had yet.
Each of these steps is reversible on its own, which is what keeps the program from turning into a rewrite. The workflow definition itself is touched exactly once, at step three, and never again.
Frequently Asked Questions
The questions below come up in nearly every architecture review where routing is on the table. They are worth settling before you write the first registry entry.
Does multi-model routing require rewriting existing agent workflows?
No. The workflow keeps its step definitions and references a logical role such as extractor or planner, while a registry binds that role to a model at runtime, so retiering a step becomes a config change.
What is the difference between a fallback chain and an escalation path?
Fallback handles availability — a 429, a 5xx, or a timeout sends the same request to an equivalent-tier model. Escalation handles difficulty, promoting a step to a stronger tier, and it should be metered separately.
Which agent steps should stay on a frontier model?
Keep frontier models on steps involving open-ended reasoning, multi-constraint planning, ambiguous tool selection, or customer-facing prose. Classification, extraction, and formatting rarely need that tier.
How do you compare cost across models with different tokenizers?
Stop metering tokens and start metering dollars per completed workflow. Tokenizers differ enough that token counts are not comparable across providers, and per-step token deltas hide retries and escalations.
What is the biggest risk when introducing model routing?
A silent downgrade — a fallback that quietly serves a weaker model during an incident and returns schema-valid but wrong output. Log the model and version actually used per step, and fail closed on write steps.
Do you need a shadow mode before moving a step to a cheaper model?
Yes. Run the candidate on live traffic without serving its output, score it against the same golden set as the incumbent, and cut over only when the per-role quality gap holds across a full traffic cycle.
Map Your Routing Layer Before You Need It
If you are running production agent workflows against a single model endpoint and trying to work out which steps can safely move tiers, the team at iSimplifyMe builds and operates multi-model agent systems across CRM, ticketing, and data warehouse environments every week. Reach out for a working session — we will tier your existing steps, build the first per-role golden set with you, and leave you with a registry design and a rollout order you can ship.
You can also start with our broader AI agent operations coverage, which walks through the adjacent problems — observability, evaluation, cost governance, and release management — that a routing layer depends on.