Automatic Retry Logic for Agent Tasks That Hit Rate Limits

Rate limits don't kill agent tasks. Missing retry architecture does. LLM API calls fail somewhere in the 1 to 5% range in production, and most of that failure has nothing to do with the model getting a wrong answer. It's throttling, timeouts, and server errors, the kind of failure that traditional software handles fine but agent systems handle badly, because a single agent request is rarely a single call.

A planner calls the model. Then a tool runs. Then a summarizer. Then a verifier checks the output. That's four or five calls sharing the same rate window, and one agent serving a handful of users can rack up hundreds of API calls a minute before anyone on the team notices. When one call in that chain fails, the whole pipeline stops, not just the step that failed.

Ten agents hit a rate limit at the same moment and all get a 429 back at the same moment. If each one retries after a flat one-second delay, they all retry at the same moment too, which produces another burst, then another wave of 429s. Picture an orchestrator running five subagents, each with its own retry logic and no awareness of the others: one rate limit event turns into 50 retry attempts inside a single second. The fleet burns API calls nonstop, makes zero progress, and racks up errors, and every one of those failed calls, even the ones that only got partway through, still gets billed.

A runaway retry loop with no exit condition is a billing event in slow motion, every failed call still gets charged, and without alerting, the damage runs for hours before anyone notices. Proper rate limit handling cuts redundant API spend by something like 40%, and that's not a rounding error worth ignoring. That's architecture, not a tuning knob.

The rate limit itself isn't the failure. The failure is what's missing around it, and most teams are staring at the wrong number.

Every provider enforces at least two independent limits: requests per minute (RPM) and tokens per minute (TPM). You can blow past either one without touching the other. Most teams only watch RPM, because it's the easier number to picture, and that habit is exactly backwards. TPM is the limit that actually breaks production, and it's the one nobody's dashboard shows by default.

A request with a 50-token prompt and a request with a 10,000-token prompt count identically against RPM, but they are nowhere close to identical against TPM. Send one huge prompt with a long context window and a long expected output, and you can exhaust your token budget while your request count still looks perfectly healthy. Then the next batch of calls comes back with 429s, and the team stares at the RPM dashboard wondering what happened. RPM was never the problem. It was never even close.

As of mid-2026, per OpenLegion's tracked limits, here's roughly where the major providers sit at their entry tier:

• OpenAI gpt-4o, Tier 1: 500 RPM, 30,000 TPM, 10,000 requests per day
• OpenAI gpt-4o-mini, Tier 1: 500 RPM, 200,000 TPM, 10,000 requests per day
• Anthropic Claude 3.5 Sonnet, Tier 1: 50 RPM, 40,000 TPM, 1,000 requests per day
• Google Gemini 1.5 Pro, Tier 1: 360 RPM, 4,000,000 TPM

Look at that Anthropic row again. Fifty agents each making one call a minute will crowd right up against that 50 RPM ceiling, and the 1,000 requests-per-day cap gets eaten faster than most teams expect, often during development and testing, well before anything reaches production.

The fix is a mindset shift, not a complicated build: estimate the token cost of a call before you send it, using something like tiktoken or a rough per-character heuristic, and draw down from a budget proactively. That's the real difference between reactive and proactive rate handling. Reactive catches the 429 after the fact. Proactive never sends the call that was going to fail in the first place. Knowing which limit you're actually hitting changes which fix you reach for, and conflating the two costs you an afternoon tuning the wrong dial.

Exponential backoff with jitter: the canonical pattern and how to parameterize it

Retrying on a fixed delay feels intuitive and is almost always wrong. If every agent waits exactly one second and tries again, they all retry at the same instant, which is exactly the storm described above, just repeating on a loop.

Exponential backoff fixes the timing. Start with a small base delay, somewhere around 1 to 2 seconds, double it on each subsequent attempt, and cap it at some sane maximum so the wait doesn't spiral into minutes. That gives the provider's system room to recover from whatever load spike triggered the throttle in the first place. Research on distributed systems from AWS puts the reduction in retry storms from exponential backoff with jitter at somewhere between 60 and 80%, big enough that skipping it should feel irresponsible.

Jitter is the piece people skip, and it's the piece that actually breaks the synchronization. Add a small random component, a few hundred milliseconds, on top of the calculated delay. Without it, a fleet of agents that all backed off from the same event will still retry in lockstep, just later. With it, the retries spread out and the burst doesn't reconstitute itself at the next window boundary.

The formula, stripped down: wait time equals the smaller of your cap or base times 2 to the power of the attempt number, plus a random jitter value. In practice that might look like 2 seconds, then 5, then 12, each one roughly double the last with some noise thrown in. Exact numbers should track your provider's actual limits, your SLA, and how urgent the workflow is.

Not every workflow should run the same schedule, either. A customer-facing agent answering someone in real time needs a tight max delay and a small number of attempts, because nobody's going to wait 40 seconds for a chatbot reply. A background reconciliation job running overnight can afford to wait minutes between tries and can afford more of them.

If a 429 response carries a Retry-After header, use it. Full stop. OpenAI's own documentation says as much. Don't calculate your own backoff when the provider has already told you exactly how long to wait; fall back to your own exponential schedule only when the header is missing or clearly bogus. Teams that build this branch in report fewer wasted attempts, simply because they stop guessing when the answer was handed to them.

Set a ceiling on attempts, too. No upper bound is its own failure mode: a retry loop with no exit is a runaway process by definition. Most teams cap it somewhere between 3 and 5 attempts; the Finance Agent Benchmark paper used 8 as its outer limit. Past that, fail the task and move on. If you're working in Python, the Tenacity library handles most of this out of the box: configure wait_exponential and stop_after_attempt, and the scheduling logic is done for you.

Classifying errors before retrying: what not to retry matters as much as how to retry

Not every failure deserves a retry. Some deserve an immediate stop, and treating the two the same is the more expensive mistake by a wide margin.

The line runs between retriable and non-retriable errors. Retriable failures are the transient kind: 429 rate limits, 500 or 502 or 503 or 504 server errors, network timeouts, a tool call that timed out because of some brief upstream hiccup. These resolve with time and another attempt.

Non-retriable errors are a different animal entirely, and retrying them wastes money at best and causes real damage at worst.

• A 429 that comes back with "quota" or "insufficient" in the error message isn't a transient throttle, it's a budget that's gone. Retrying won't fix it. That needs a monitoring alert and a graceful fallback, not another attempt.
• Authentication errors, 401s and 403s, fail identically every single time with the same credentials. Retrying just burns a call to relearn what you already knew.
• A 400 means the request itself is malformed. Sending the same malformed request again produces the same 400.
• Anything with a side effect, charging a card, sending a message, updating a record, needs idempotency controls before it gets anywhere near a retry, or you risk doing the thing twice.

There's a subtler failure buried in here too, and it's the one worth losing sleep over. If an LLM only sees a synthetic error message where a tool response should have been, it doesn't know it just hit a rate limit. It sees a gap, and because it wasn't trained on that specific failure signal, it may paper over the gap with a creative but wrong workaround. In financial workflows, that's how a rate limit turns into a hallucinated transaction, and the cost of that mistake dwarfs whatever the API call itself would have cost. Get the classification wrong here and nothing downstream matters.

A working retry policy answers four questions before it fires again: which errors get retried automatically, how long to wait, when to give up and escalate to a human, and what gets written down before the next attempt goes out. Two checks drive most of that branching logic: does the response carry a Retry-After header, and does the error body say quota exhaustion or transient throttling. Everything downstream follows from those two answers.

Moving rate governance out of individual agents and into shared infrastructure

Each agent watching its own rate limit sounds reasonable until you remember that none of them can see what the others are doing. Application-layer self-policing doesn't solve a shared quota problem, because the quota isn't per-agent, it's per-account, and no agent has visibility into the account-wide picture. Most teams get the architecture backwards here: they harden each agent instead of building the layer above them, and hardening the wrong layer is worse than doing nothing, because it buys false confidence.

The fix is to pull rate governance out of every individual agent and put it into one shared layer sitting in front of the provider. In practice, that means a single token bucket shared across the whole fleet, with jitter baked in, replacing a pile of uncoordinated per-agent retry loops with one queue that drains in a predictable, orderly way.

The token bucket algorithm, standardized in IETF RFC 2697 and 2698, is the right primitive here. A bucket holds tokens, refilled at a steady rate up to some cap. Every API call draws tokens out of the bucket, and calls that carry more estimated input weight draw more tokens. When the bucket's empty, the call blocks and waits for a refill, instead of firing off and failing. That shift, blocking before the call instead of catching the failure after, is what turns reactive rate limiting into proactive rate limiting. It also mirrors how providers actually behave, allowing bursts up to the RPM ceiling rather than metering every call at a perfectly even pace.

On top of the bucket, three signals should trip a breaker at the fleet level, not just at the individual agent level:

• Cost velocity. If a user or agent is burning tokens well above its rolling average, it's probably stuck in a loop, not doing legitimate work.
• Repeated prompts. The same or a near-identical prompt firing several times in a short window is a bug signature, not a sign of intent.
• Error rate. If 20% of an agent's calls are coming back 4xx in a rolling window, stop sending and surface the problem instead of quietly absorbing it.

Pre-flight quota checks help too. If a workflow needs six API calls to finish, check that quota exists for all six before making the first one, rather than getting stranded halfway through. Set alerts for when remaining budget drops below 10%, so someone finds out before the bucket runs dry, not after.

Coordinating all of this across a multi-agent fleet needs a shared store, Redis or an equivalent database, that every agent checks before it fires a call. Without that shared state, the math doesn't work: 10 agents each making one call a second against an OpenAI Tier 1 account will burn through the 500 RPM ceiling in 50 seconds flat, and no amount of per-agent care prevents it. Per-agent care was never where the problem lived, and it never will be.

Circuit breakers: when to stop retrying entirely and protect the rest of the system

Backoff and jitter smooth out transient failure. They don't help when the provider itself is genuinely down, and that's where a circuit breaker earns its keep.

A circuit breaker runs through three states. Closed is normal operation, calls go through as usual. Open means failures have crossed a threshold, and the breaker stops attempting calls entirely, failing fast instead of wasting tokens on requests that are almost certainly doomed. Half-Open is the recovery test: after a cooldown period, one request goes out to check whether the dependency has come back. Succeed, and the circuit closes again. Fail, and it reopens.

Watch the failure rate for whatever you're depending on, the LLM provider itself in most cases. Cross the threshold within the monitoring window, open the circuit. While it's open, every call fails immediately without ever reaching the network, so there's no cost and no wasted latency. Multi-agent systems fail somewhere between 41% and 86.7% of the time in production when nobody's built deliberate fault tolerance into them, and that range alone should settle the argument for whether resilience engineering deserves the same attention as the agent's core reasoning logic.

Every retry that doesn't succeed adds cost without adding progress. Once a step has failed some fixed number of times in a row, the default should be to stop and escalate, not to keep trying.

Beneath the circuit breaker sits a fallback chain: the primary model, then a cheaper or faster alternative if the primary's unavailable, then a semantic cache that returns a stored answer for a semantically similar past query, and finally a graceful failure response if none of that works. That chain should live in one place, configured once, so every agent inherits the same fallback behavior instead of each one reinventing its own version.

And at the very end of the chain sits a human. Some failures just can't resolve themselves. After some fixed number of retries, the system should open a ticket or a notification for a person and pause the workflow, particularly for document processing, invoice generation, or anything where being right matters more than being fast. Stacked together, the layers read cleanly. Backoff handles the transient stuff, the circuit breaker handles a provider that's actually down, the fallback chain handles a model that's unavailable, and human escalation catches whatever's left.

Idempotency and durable state: making retries safe to execute

Say a process crashes after finishing step 8 of a 10-step task. Retry the whole thing from scratch and you risk duplicating whatever side effects already happened at steps 1 through 8, burning tokens redoing work that already succeeded, and throwing away intermediate state that took real compute to produce.

Idempotency is the property that makes retrying safe: an operation that produces the same result whether it runs once or ten times. Build for it, and a retry never has to ask whether the previous attempt already succeeded, because running it again causes no harm either way.

For agent workflows specifically, the risk without idempotency is concrete. Charging a customer twice. Sending the same message twice. Writing conflicting versions of the same record. All of that is what happens when a retry fires without any memory of what already happened. Idempotency keys solve it: assign a unique key to an operation before the first attempt, and if a retry comes in carrying that same key, the receiving system recognizes it and returns the original result instead of executing the action again.

That only works if progress gets written down somewhere durable, a database or an external store that survives a crash. Checkpoint after every successful step. On recovery, read the checkpoint back, skip whatever's already done, and resume from there rather than starting over.

One detail decides whether this actually holds up under a crash: state has to be written before the next call goes out, not after it comes back successful. Write it after, and a crash between the call and the write loses the checkpoint entirely, defeating the whole point.

In financial and legal workflows, none of this is optional. Audit requirements mean the system has to prove a compensating action was taken. When an action can't be undone, a payment that's already gone through, the recovery path requires manual intervention rather than an automatic rollback. That's a compensating transaction, with its own approvals and its own paper trail, handled deliberately rather than automatically.

Durable execution frameworks: infrastructure that handles retry and recovery automatically

Through 2025 and 2026, the industry has converged on three patterns that work together rather than compete: durable execution, which persists state and handles retry at the infrastructure layer automatically, finite state machines, which model an agent's states and error paths explicitly instead of leaving them implicit in code, and event-driven orchestration, which coordinates work asynchronously across decoupled components.

Hand-building all of the above into every agent is the wrong default once a fleet grows past a couple of workflows. Temporal has become the leading platform built specifically for durable agent execution, handling the state persistence and retry logic that would otherwise need to be built by hand into every agent, using the patterns already laid out above: backoff, jitter, idempotency, checkpointing. This infrastructure exists so individual teams don't each rebuild the same retry engine badly, and given how often that engine gets rebuilt badly anyway, that's not a small thing to offer.