Do you know how many tokens per minute your production agents draw at 2 p.m. on the last business day of the quarter? If you own the platform those agents run on, you should.
Many teams find out the hard way — when a claims agent stalls at step six of nine, a provider returns a wall of 429s, and a half-finished workflow sits holding a lock on a Salesforce record. That said, the root cause in most of these incidents is a capacity plan that was never written.
Frontier-model access now behaves like any other constrained supply. Quotas are set by the provider, tier increases depend on spend history and review queues, and pricing, terms, and model availability can all move within a single quarter.
This piece covers how ops leaders size token quotas, reserve throughput, and set per-workflow priority tiers before a provider limit stalls an agent mid-process. Protecting the downstream systems an agent calls is a separate discipline covered in agent backpressure for downstream APIs, and switching models under failure is covered in agent fallback design.
Why Provider Rate Limits Become A Capacity Planning Problem
During a pilot, rate limits are an afterthought. A handful of users trigger a handful of runs, and the default quota on Bedrock or the Anthropic API absorbs the load without anyone checking a dashboard.
Rate limits become a capacity problem once agents run in production, because each run fans out into many model calls. A workflow that fits the quota at pilot volume can stall at 10x volume with no code change.
The math changes at production volume because an agent is a loop rather than a single call. Each run plans, calls tools, reads results, and calls the model again, so one user action can become eight, twelve, or twenty model invocations.
What's more, those invocations get heavier as the run progresses. Tool output and conversation history accumulate in the context window, which means step nine of a run often sends several times the input tokens of step one.
Keep in mind that the provider does not see your workflows. It sees an account, a region, and a model ID drawing against a shared ceiling — and every agent in the organization competes for that same ceiling unless you divide it deliberately.
What Does A Production Agent Actually Consume?
Before anyone can size a quota, the team needs a per-workflow consumption profile. Providers typically meter three dimensions at once, and each one can become the binding constraint:
- Requests per minute (RPM). The count of model calls, regardless of size. Chatty agents that make many small tool-routing calls hit this ceiling first.
- Input tokens per minute (ITPM). The volume of prompt, context, and tool output sent to the model. Long-running agents with growing context windows usually exhaust this one first.
- Output tokens per minute (OTPM). The volume the model generates. Agents that draft documents, write code, or produce long structured outputs press against this limit.
Some providers combine input and output into a single tokens-per-minute figure, while others meter them separately. Either way, the operating ceiling is whichever dimension the workflow reaches first.
An agent draws on requests, input tokens, and output tokens per minute at once. Whichever ceiling it hits first sets its real capacity, and growing context usually makes input tokens the binding limit.
A Worked Example: An Invoice Reconciliation Agent
Consider an invoice reconciliation agent that averages nine model calls per run, with roughly 6,000 input tokens and 600 output tokens per call. That works out to approximately 54,000 input tokens and 5,400 output tokens per completed run.
At month-end, the finance team wants 40 runs in flight at once, and each run takes about three minutes. That is roughly 13 runs completing per minute — approximately 720,000 input tokens, 72,000 output tokens, and 120 requests every minute at the typical run size.
However, the typical run is the wrong number to plan against. In this example the P95 run consumes about 2.2 times the typical token count, because exceptions trigger extra lookups and retries, which pushes peak input demand toward 1.6 million tokens per minute.
A team that sized its quota at 750,000 input tokens per minute would pass every pilot test and still stall on the first busy close. This is why the measured consumption profile, rather than the vendor's default tier, is where capacity planning starts.
How Do You Size Token Quotas For Agent Workflows?
Sizing is arithmetic, but it is arithmetic on measured numbers rather than estimates from a design doc. Here is the sequence we run for each production workflow:
- Instrument tokens per step. Log input, output, and cached tokens for every model call, with the workflow ID, step name, and pinned model version attached. Without per-step data, you cannot tell which step drives the peak.
- Build the P50 and P95 run profiles. Use at least two weeks of production or shadow-mode traffic. The P95 run sets the quota, and the P50 run sets the budget forecast.
- Model peak concurrency. Pull it from the business calendar — month-end close, open enrollment, a product launch — rather than from last week's traffic. The busiest hour of the busiest day is the planning target.
- Multiply and add headroom. P95 tokens per run times peak runs per minute gives the floor. We add 30-40% headroom on top for retries, prompt drift after a model-version change, and the spike nobody forecast.
- Sum across workflows that share a quota. Every agent drawing on the same account, region, and model ID shares one ceiling. The plan has to add them up and check whether their peaks coincide.
Size quotas from the P95 run, not the average. Multiply P95 tokens per run by peak runs per minute, then add 30-40% headroom for retries, prompt drift, and the volume spike nobody forecast.
Caching changes the arithmetic in your favor. On several providers, cached prompt prefixes are billed at a steep discount and can count differently against input-token limits, so a stable system prompt and tool catalog at the front of every call recovers real capacity.
Routing works from the other direction, because sending classification and extraction steps to a smaller model frees frontier-model quota for the reasoning steps that need it. Together, caching and routing are how teams keep AI spend flat while token usage grows, and the routing logic itself is covered in model routing for production agents.
Reserved Throughput Versus On-Demand Capacity
Once the team knows its peak, the next decision is where that capacity comes from. Each major provider offers some version of the same options, under different names and commercial terms.
On AWS, Bedrock offers on-demand inference against account-level service quotas, cross-region inference profiles that spread on-demand traffic across regions, and Provisioned Throughput purchased in model units. Azure OpenAI sells provisioned throughput units (PTUs), and the direct OpenAI and Anthropic APIs offer usage tiers, batch endpoints, and priority capacity arrangements.
The tradeoffs line up as follows:
| Capacity Type | Best Fit | Cost Behavior | Primary Risk |
|---|---|---|---|
| On-demand (default quota) | Spiky or unpredictable load, early production | Pay per token, no idle cost | Shared ceiling, 429s at peak |
| Cross-region on-demand | Moderate peaks within one geography | Pay per token, traffic may route across regions | Data residency review, still bound by a quota |
| Reserved / provisioned | Steady load, hard deadlines, Tier 0 and Tier 1 | Hourly or term commitment, billed whether used or not | Idle spend, bound to one model version |
| Batch | Backfills, evaluation runs, re-embedding | Roughly half the real-time price on major APIs | Results in hours, up to 24 hours |
Cost honesty matters here. At an illustrative $40 per model-unit hour, one reserved unit runs about $29,200 a month whether the agents call it or not — a sound purchase for a claims workflow running at 85% utilization around the clock, and an expensive one for a workflow running at 20%.
Be aware that reserved capacity is usually bound to a specific model version. When that model is deprecated, the reservation has to move with it, which is why capacity planning and model deprecation migration belong on the same calendar.
Reserve throughput for workflows with steady load or hard deadlines, and leave spiky or deferrable work on on-demand and batch capacity. Reserved units bill hourly whether or not an agent calls them.
Why Priority Tiers Belong In The Workflow Definition
When demand exceeds supply, something has to wait. Without tiers, that decision is made by whichever agent happens to send its request first — and the month-end close competes on equal footing with a nightly CRM enrichment job.
We assign every production workflow a tier at design time and record it in the workflow definition alongside its agent SLOs. A typical four-tier scheme looks like this:
- Tier 0 — Customer-facing, synchronous. A Zendesk support agent answering a live chat, where time-to-first-token is the user experience. These workflows hold a reserved floor that lower tiers can never borrow.
- Tier 1 — Deadline-bound. Claims adjudication, month-end reconciliation, and regulatory filings. These hold a reserved floor and may borrow unused Tier 2 and Tier 3 capacity.
- Tier 2 — Internal asynchronous. CRM enrichment, ticket triage, and document summarization for internal teams. These run on shared on-demand quota and queue when it tightens.
- Tier 3 — Deferrable. Backfills, evaluation runs, and re-embedding jobs. These move to batch endpoints by default and are shed first under pressure.
The tiers are enforced by a token broker — a small admission-control service that holds a ledger of available tokens per provider, model, and region, typically as a token bucket in Redis or DynamoDB. Every agent step asks the broker for a grant before it calls the model, and the broker answers based on the tier floor, current burn, and remaining headroom.
Note that the broker governs the agent's inbound supply of model capacity. Rate-limiting the agent's outbound calls to Salesforce, ServiceNow, or Workday is a different control with a different owner, and it should be designed as one.
Priority tiers decide which workflows keep their tokens when the quota runs short. Customer-facing and deadline-bound agents hold a reserved floor, while batch and backfill work is shed first and resumes later.
What Happens When An Agent Hits A Limit Mid-Process?
Even a well-sized plan will meet a 429 eventually. A provider incident, a regional capacity crunch, or an unforecast spike can push any workflow past its ceiling, so the real question is what the agent does at step six of nine.
A naive retry loop is the worst available answer. It burns the quota other tiers need, it can restart the run from the top and double the token spend, and it can repeat a write that already landed in the system of record.
- Checkpoint after every completed step. Persist run state — inputs, tool results, and the step pointer — to DynamoDB or the state store behind your Step Functions state machine. A resumed run reads the checkpoint instead of replaying the conversation.
- Honor the provider's retry signal. Most providers return a retry-after or rate-limit reset header alongside a 429. Wait at least that long, add jitter, and cap attempts per step.
- Attach idempotency keys to every external write. If the step that hit the limit had already posted to the ERP, the resumed run must not post again. The key makes the second attempt a no-op.
- Park the run once retries are exhausted. Past the retry cap, move the run to an agent exception queue with its checkpoint attached. It resumes when the broker grants capacity again.
These are the same retry policies, checkpointing, and incident patterns we apply to regulated AI workloads. Accordingly, the capacity plan and the reliability design should be written together, by the same team, in the same review.
When an agent hits a limit mid-run, it should checkpoint state, honor the provider's retry signal, and resume from the last completed step. Restarting from step one doubles token spend and risks duplicate writes.
Why Supply Risk Belongs In The Capacity Plan
Up to this point, the plan assumes the provider's terms hold still. In practice, frontier-model access can tighten for reasons unrelated to your traffic — compute scarcity during a major model launch, a price change, revised usage policies, a regional outage, or a retirement date that lands mid-quarter.
Accordingly, the capacity plan should name its supply assumptions explicitly and test what happens when each one fails. We track four:
- Quota concentration. What share of Tier 0 and Tier 1 capacity depends on a single provider, model, and region? Anything above roughly 70% deserves a documented second source.
- Increase lead time. How long did the last quota increase take to approve? File the next request a full quarter before the volume event that needs it.
- Price exposure. What happens to the monthly bill if the per-token price on the primary model rises 25%? Name the dollar figure so finance is not surprised.
- Degraded-mode quality. If Tier 0 had to run on the fallback model for a week, would it still meet its SLO? Run the evaluation suite against the fallback before you need it.
This is where capacity planning meets fallback design, because a fallback path is only real if it has quota of its own. The secondary provider needs a provisioned account, tested credentials, and a sized limit before the primary goes dark — the same discipline behind treating the model as a dependency with a retirement date.
A fallback model only counts once it has its own approved quota, tested credentials, and a passing evaluation run against the workflows it is meant to carry.
How Do You Know Your Capacity Plan Is Working?
A capacity plan remains a hypothesis until production traffic tests it. The signals worth putting on the platform team's dashboard include:
- Quota utilization at P95, per tier. Sustained utilization above roughly 75% of the ceiling means the headroom is gone. File the increase before the next peak, not after.
- 429 rate per workflow. Any 429s on Tier 0 are an incident. A steady trickle on Tier 3 is the tiering working as designed.
- Queue age by tier. Track how long Tier 2 and Tier 3 runs wait for a broker grant. Measure it against each workflow's SLO rather than a global target.
- Resume success rate. This is the share of throttled runs that resume from checkpoint and finish without a duplicate write or a human touch. A falling rate points to checkpoint gaps.
- Reserved-unit utilization. Below roughly 50% for a sustained month, the reservation is costing more than it protects. Either move more Tier 1 work onto it or step it down at renewal.
Review these monthly alongside the business calendar. After all, the next volume spike is usually visible weeks in advance — the open enrollment window, the product launch, the quarter close — and the quota request has to land before it does.
Frequently Asked Questions
The questions below cover the edge cases that surface once a capacity plan meets production traffic.
What is the difference between TPM and RPM limits for agents?
Requests per minute caps how many model calls you make; tokens per minute caps how much volume they carry, often split into input and output. Agents with long, growing context usually hit the token limit first, while chatty tool loops hit the request limit.
Is provisioned throughput worth it for a single agent workflow?
Usually only if that workflow runs near-constant load or carries a hard deadline. Reserved units bill by the hour whether used or not, so a spiky workflow averaging 20% utilization pays for idle capacity most of the month.
How long does a provider quota increase take to approve?
Plan for days to weeks, not hours. Bedrock increases go through Service Quotas review, and direct API tiers typically step up with spend history, so file requests a full quarter ahead of a known volume event.
Do batch APIs count against real-time rate limits?
On the major direct APIs, batch jobs run against separate batch limits and are priced at roughly half the real-time rate, with results returned within 24 hours. That makes batch the natural home for Tier 3 backfills and evaluation runs.
Should agents retry immediately after a 429 response?
No. Honor the retry-after or reset header, add jitter, and cap attempts per step. Past the cap, checkpoint the run and park it in an exception queue rather than looping and burning quota other tiers need.
Build Your Capacity Plan Before The Next Volume Spike
Provider limits will tighten again, whether through a busy launch week, a price move, or a model retirement. The teams that keep agents running are the ones that measured their consumption, tiered their workflows, and secured a second source before they needed it.
Capacity planning is one discipline within a broader AI agent operations practice, and it works best when it is owned by the same team that owns routing, fallback, and reliability. Plan it as part of that whole, and the quota becomes a number you manage rather than a surprise you absorb.
If you're scaling agents past pilot volume and want a second set of eyes on capacity, the team at iSimplifyMe builds and operates production agent systems on Bedrock and direct model APIs every week. Reach out for a working session — we'll profile your token consumption per workflow, draft your priority tiers and broker design, and leave you with a quota request and reservation plan sized to your next peak.