I once watched a perfectly healthy-looking AI service lose a conversation without ever returning a 5xx. The primary provider slowed down, the gateway moved the request to a backup, and the backup returned HTTP 200. The dashboard celebrated availability. The user saw an assistant that had forgotten the last six turns.
That incident changed how I think about multi-model systems. Adding three providers behind one API is not resilience. It is only optionality. Resilience appears when the system can change its route without violating the task’s capability contract, privacy policy, conversation state, latency budget, or quality bar.
This is the part that generic “LLM fallback” diagrams usually hide. Provider rotation has at least two independent decisions:
- Which model family should perform the task?
- Which provider endpoint should serve that model right now?
Those decisions should not be collapsed into a single round-robin loop. The model router protects capability and product behavior. The provider selector adapts to capacity, rate limits, regional health, price, and policy. A state layer preserves what the next provider needs to know. A quality gate decides whether the result is safe to accept.
This article is a companion to Model Router for AI Agents, but it focuses on the problem that begins after the router has chosen a model: how to rotate providers and models under pressure without creating a new failure mode.
The thesis: A safe fallback is not “try the next API.” It is a bounded, observable state transition between capability-compatible routes, governed by retry budgets, provider health, and an explicit continuity contract.
Provider rotation is not model fallback
A provider and a model are different dimensions. The same model family may be available through a model vendor, a cloud-hosted endpoint, a regional deployment, or a gateway with its own capacity and privacy controls. A provider can be unhealthy while the model family remains the right choice. Conversely, every provider for the preferred model can be healthy while the model itself is the wrong tool for a long-context or structured-output task.
OpenRouter documents this distinction directly: provider routing can try available providers for the requested model, while model fallbacks move to a different model when the first model’s providers fail or refuse to answer. The distinction matters because a provider switch should usually preserve the model contract, while a model switch may change tool behavior, context capacity, reasoning style, output format, or safety characteristics.
| Decision layer | Question | Safe default |
|---|---|---|
| Capability route | What kind of model can complete this task? | Keep a pre-validated model family or capability class. |
| Provider route | Which endpoint can serve that capability now? | Prefer healthy, policy-approved providers with enough quota. |
| Session route | Where does this conversation or workflow keep its state? | Keep an affinity lease unless health or policy requires a move. |
| Recovery route | What should happen after an uncertain or partial result? | Reconcile state before retrying; do not assume a timeout means no side effect. |
| Quality route | Is the returned answer acceptable for the business task? | Validate structure, tools, evidence, and outcome—not only HTTP status. |
A common design mistake is to place all five decisions in a single fallbacks: [...] array. That array may be convenient, but it is not a reliability policy. A route must explain why a candidate was eligible, what failure triggered the transition, and what invariants must remain true after the transition.
The visual distinction is worth keeping in the architecture documentation because it prevents a very expensive ambiguity: are we moving traffic, or are we changing what the assistant is capable of doing?
The provider switch should preserve a capability envelope
A provider-neutral interface is useful only when it normalizes the differences that can safely be normalized and exposes the differences that cannot. “Chat completion” is not a complete contract. A production task may require tool calling, strict JSON, vision, a context window above a threshold, a specific reasoning mode, a region, a retention policy, or a known refusal behavior.
Create a capability envelope for every route. It can be stored as configuration and tested in CI, but it should also appear in the runtime decision record:
{
"capability_class": "support_tool_agent_v2",
"required": {
"tool_calling": true,
"structured_output": "strict-json",
"context_tokens": 24000,
"streaming": true,
"region": "eu-approved"
},
"preferred_model_family": "reasoning-balanced",
"allowed_provider_pools": ["primary-eu", "backup-eu"],
"fallback_policy": "provider-only-before-model-change"
}The envelope does not promise that two providers behave identically. It says that both candidates have passed the minimum contract tests for this task. That is a much more honest abstraction than pretending every OpenAI-compatible endpoint is behaviorally interchangeable.
For example, a provider may accept a JSON schema but still emit arguments that fail your parser. Another may support tools but serialize parallel calls differently. A third may have enough context capacity for the prompt but not enough for the model’s hidden reasoning or output. These differences belong in route metadata, not in tribal knowledge inside a retry handler.
Rotate on evidence, not on a timer
Blind rotation looks simple: send request one to provider A, request two to provider B, and continue cycling. It can spread traffic, but it ignores the fact that providers have different failure domains and rate-limit dimensions. A clock-based rotation can also create synchronized bursts exactly when the system is already under pressure.
The selector should combine at least five signals: admission, health, latency, policy, and recent quality. The values do not need to be perfect. They need to be recent enough to avoid routing based on a provider that was healthy ten minutes ago.
| Signal | What to observe | How it changes rotation |
|---|---|---|
| Admission | Remaining request/token budget, concurrency, queue depth | Do not send traffic to a provider that cannot admit the request. |
| Health | 5xx, timeouts, connection errors, 429 rate, circuit state | Reduce or open the provider pool when failures persist. |
| Latency | Time to first token, total latency, p95/p99 | Prefer the route that can meet the request’s tail budget. |
| Policy | Region, retention, data class, tenant restrictions | Remove ineligible providers before scoring them. |
| Quality | Schema validity, tool success, evidence checks, business outcome | Avoid routes whose “successful” responses require repairs. |
OpenAI’s rate-limit guidance recommends honoring Retry-After when present, adding jitter, bounding attempts and total retry time, and not retrying quota or billing errors that require action. Anthropic exposes request, token, input-token, and output-token remaining/reset signals, and also warns that sudden traffic acceleration can hit a separate limit. A provider rotation layer should normalize these signals into a common admission interface while preserving the raw headers for diagnosis.
One useful mental model is AIMD admission. Add capacity gradually after success; multiply admission down after a rate-limit or overload signal. Sierra describes a similar congestion-aware selector to avoid oscillation between providers, including priority-aware shedding when capacity is constrained. The exact coefficients are product-specific, but the principle is portable: recovery should be gradual, not a synchronized flood.
A simplified decision function might look like this:
def eligible(route, request, now):
return (
route.policy_allows(request.data_class, request.region)
and route.supports(request.capability_envelope)
and route.circuit.is_closed_or_probe_allowed(now)
and route.admission.can_accept(request.estimated_tokens)
)
def score(route, request):
return (
0.30 * route.health_score
+ 0.25 * route.quality_score(request.task_kind)
+ 0.20 * route.latency_score(request.latency_budget_ms)
+ 0.15 * route.capacity_score(request.estimated_tokens)
+ 0.10 * route.affinity_score(request.session_id)
)
candidates = [r for r in routes if eligible(r, request, now)]
selected = max(candidates, key=lambda r: score(r, request)) if candidates else NoneThe weights are not universal defaults. In a voice assistant, tail latency may dominate. In an invoice extraction workflow, structured-output validity and evidence checks may be worth more than a fast first token. The important property is that the policy is named, versioned, and measurable.
Circuit breakers need half-open probes, not permanent exile
A circuit breaker protects a provider pool from being hammered while it is failing. It should not become a permanent ban caused by one transient timeout. A practical provider circuit has three states:
| State | Behavior | Transition |
|---|---|---|
| Closed | Admit traffic according to the normal score. | Open after a threshold of classified failures. |
| Open | Reject normal traffic locally and choose another eligible pool. | Move to half-open after a cool-down. |
| Half-open | Permit a small, controlled probe budget. | Close after healthy probes; reopen after failure. |
The failure classifier matters. A connection reset, timeout, 429, 401, context overflow, content refusal, schema error, and business validation failure do not mean the same thing. A 401 usually needs credential or configuration action. A context overflow may be recoverable by compaction or a larger-context model, not by repeating the same request on another provider. A schema failure may point to capability drift rather than provider health.
The drawing is intentionally operational rather than decorative: the breaker, admission budget, and backoff policy are three different controls. Combining them into one “retry” switch makes incidents harder to contain.
Use separate breakers for separate failure domains. One provider’s embedding endpoint should not open the breaker for its chat endpoint. One tenant’s quota exhaustion should not eject the provider for every tenant. One model family’s structured-output regression should not be hidden as a transport outage.
Retry budgets prevent rotation from becoming a retry storm
The most dangerous fallback implementation is a loop that retries every error against every provider. If 100 requests fail together and each tries three providers five times, the system can produce 1,500 attempts while the underlying outage is still happening. The fallback traffic becomes the incident.
A retry policy needs three budgets:
- Attempt budget: the maximum number of provider/model attempts for one logical request.
- Time budget: the maximum wall-clock time before returning, queuing, or asking for human recovery.
- Blast-radius budget: the maximum additional load any single provider pool may receive from failover traffic.
The client should respect a provider’s Retry-After when valid, then add bounded jitter. If the header is missing, use exponential backoff with a cap. Do not let each application layer add its own retries independently; compose the SDK, gateway, queue, and agent-runtime budgets into one decision.
async def call_with_rotation(request, route_plan):
deadline = monotonic() + request.time_budget_ms / 1000
attempts = 0
last_error = None
for route in route_plan:
if attempts >= request.max_attempts or monotonic() >= deadline:
break
if not route.admission.reserve(request.estimated_tokens):
continue
attempts += 1
try:
response = await route.call(request, timeout=deadline - monotonic())
result = validate_response(response, request.contract)
if result.ok:
return result
last_error = result.error
if not result.retryable_quality_failure:
break
except ProviderError as error:
last_error = error
route.record(error)
if not error.retryable:
break
await bounded_backoff(last_error, deadline)
return recover_or_escalate(request, last_error)Notice what this code does not do. It does not retry after a side effect merely because the response timed out. It does not treat every 429 as permission to hammer a fallback. It does not retry a schema mismatch forever. It makes the logical request, the route plan, and the stopping conditions explicit.
For a tool-using agent, pair this with the idempotency contract described in Idempotent AI Actions. A provider switch is safe only if the request can be replayed without duplicating an external action, or if the runtime can reconcile the action’s outcome before replaying it.
Stateful failover: availability is not continuity
A fallback can return a response while still losing the task. This is especially visible in chat and voice systems, but the same problem exists in multi-step agents. If the fallback provider receives only the latest user message, it may not know the plan, tool results, constraints, or decisions that shaped the current turn.
ContinuityBench frames this as a measurable distinction between availability and conversational continuity. Its paper proposes forwarding enough history to reconstruct state across heterogeneous endpoints and reports a 99.20% Continuity Preservation Rate in its own evaluation of 750 failover events. That result is useful as evidence that continuity can be measured; it is not a guarantee that any implementation will achieve the same number.
The important object in this diagram is not the arrow between providers. It is the state hash and the immutable event trail that make the arrow safe to follow.
A stateful failover design should define a continuity envelope:
| State component | Minimum question before switching |
|---|---|
| Conversation history | Does the next provider receive the relevant turns, not necessarily the entire transcript? |
| System policy | Are system/developer instructions preserved with version and integrity metadata? |
| Tool state | Are tool results and pending actions represented as immutable events? |
| Working memory | Which summaries, retrieved evidence, and constraints are still valid? |
| Output boundary | Has any partial stream already reached the user? |
| Route identity | Is the new provider allowed to see the state under tenant and region policy? |
Forwarding the entire transcript is not always the right answer. It can increase latency and leak irrelevant data. A safer design stores a canonical event history, then builds a provider-specific context projection with a stable state hash. The projection can be compacted, but the runtime should be able to explain which events were included and which were intentionally omitted.
A provider switch after streaming begins needs its own rule. If the user has already seen half a sentence, silently continuing with a different model can create a visible tone or factual discontinuity. Stop the stream and retry only when the product can mark the boundary, or keep the original route sticky until the turn finishes. Sierra makes the same practical point: switching after user-visible streaming has begun may be inappropriate when it changes behavior or consistency.
Model rotation needs quality gates, not only health checks
Health checks answer, “Can I get a response?” They do not answer, “Did the model complete the task correctly?” A model may be online while its tool-call arguments drift, its structured output becomes less stable, or its refusal behavior changes after a provider-side update.
Every fallback candidate should have a route-specific quality gate. The gate might verify JSON schema, required fields, citation presence, tool-call validity, policy labels, or a business invariant. For an agent, also verify that the correct tool was selected and that the next state transition is legal.
| Route type | Minimum acceptance gate |
|---|---|
| Extraction | Schema validation, field confidence, and source-span checks. |
| Tool call | Tool name, argument schema, authorization, and idempotency key. |
| Retrieval answer | Evidence coverage, freshness policy, and unsupported-claim check. |
| Planning | No forbidden action, bounded step count, and valid dependencies. |
| User-facing response | Safety policy, coherent state, and no unexplained partial-stream boundary. |
This is where shadow traffic and canaries become more valuable than a provider status page. Send a bounded, privacy-safe sample to a candidate route without allowing it to execute tools or mutate state. Compare task outcomes, not just token cost and response preference. The existing agent regression suite is the right place to store these route-pair tests.
A route change should be reversible. Keep the route policy version, candidate set, provider, model, error class, validation result, and final business outcome in a privacy-aware record. The agent observability article describes how to trace prompts, tool calls, tokens, and cost without turning the trace into a second data leak.
Route leases stop session flapping
If the selector chooses independently on every turn, a single conversation can bounce between providers. That makes latency noisy, invalidates prompt caching, complicates debugging, and can change the assistant’s voice. The opposite extreme—permanent stickiness—wastes healthy capacity and makes recovery slow.
Use a route lease. The lease binds a session or workflow to a capability-compatible model/provider pool for a short period or a bounded number of steps. It can be renewed after healthy outcomes and revoked when the circuit opens, the policy changes, or the route violates its quality SLO.
session_id -> capability_class -> route_pool -> lease_expiry -> state_hashThe lease should not contain secrets or raw prompt text. It is a routing hint backed by policy. When the lease is revoked, the new route must receive a continuity envelope and a reason code such as provider_429, p95_latency_breach, policy_change, or quality_regression.
What to measure after rotation
A provider rotation project is not successful because the error rate went down. It is successful when the system survives failure without creating a larger, less visible failure elsewhere.
Track the following as a joint outcome rather than separate dashboards:
| Metric | Why it matters |
|---|---|
| Provider failover rate | Shows how often the preferred dependency is unavailable or constrained. |
| Failover success rate | Separates a route transition from a merely returned HTTP response. |
| Continuity Preservation Rate | Measures whether conversation/workflow state survived the switch. |
| Retry amplification | Additional attempts per logical request during incidents. |
| Route flapping rate | Number of route changes per session or workflow. |
| Cost per successful outcome | Includes retries, repairs, escalations, and tool loops. |
| Quality delta by route pair | Detects silent behavior changes between primary and fallback. |
| p95/p99 recovery latency | Captures the tail users feel during provider incidents. |
| Policy rejection rate | Shows whether candidate filtering is too late or too permissive. |
The key denominator is usually logical requests, not HTTP attempts. A system that returns 99% HTTP 200 after tripling its attempts may be less reliable and more expensive than a system that fails fast and asks for clarification.
A practical rollout sequence
Start with a static, human-readable route matrix. List each task class, its required capability envelope, its approved provider pools, its model fallback policy, its maximum attempts, and its terminal behavior. Do not start with a machine-learned router when the team cannot explain a route decision.
Next, add provider health and admission signals without changing traffic. Observe rate-limit headers, queue depth, p95 latency, and error classes. Then run contract probes that exercise tools, structured output, context limits, streaming, and privacy filters. A provider is not “ready” merely because its /health endpoint returns 200.
After that, introduce route leases and bounded provider-only failover. Keep model fallback disabled until the team has tested capability equivalence. Run shadow evaluations against candidate model routes, then canary only the task classes with clear quality gates. Finally, chaos-test a provider outage, a partial stream failure, a 429 storm, a stale credential, a context overflow, and a state-reconstruction bug.
The safe order is deliberate:
| Stage | What changes | What must be proven |
|---|---|---|
| Contract | Capability envelopes and provider adapters | Equivalent minimum behavior is testable. |
| Admission | Health, quota, latency, and circuit state | Bad routes are filtered before sending traffic. |
| Recovery | Retry budgets and provider-only failover | One logical request cannot create an unbounded storm. |
| Continuity | State projection and route leases | A switched conversation preserves the task state. |
| Quality | Model fallback and shadow/canary gates | A different model does not silently lower outcomes. |
| Optimization | Learned scores, price/latency tuning | The policy remains explainable and reversible. |
The advanced checklist
Before rotating models or providers in production, I would ask:
| Question | Evidence I want |
|---|---|
| Are provider rotation and model fallback separate? | Distinct policy layers and route records. |
| Can every candidate satisfy the capability envelope? | Contract tests for tools, JSON, context, streaming, and policy. |
| What does a 429 mean here? | Raw headers, normalized admission state, and a bounded response. |
| Can failover preserve state? | A continuity envelope, state hash, and reconstruction test. |
| Can the system stop retrying? | Attempt, time, and blast-radius budgets. |
| Can a provider recover gradually? | Half-open probes, AIMD-style admission, and priority shedding. |
| Can we detect silent quality drift? | Route-pair evals, shadow traffic, and business-outcome metrics. |
| Can we explain a route change six hours later? | Privacy-safe decision records with policy version and reason code. |
The goal is not to make the system indifferent to providers. Providers differ, and those differences are useful. One may have stronger tool calling, another may have better regional capacity, and a third may be valuable as an emergency route. The goal is to make the differences explicit enough that the system can use them without surprising the user.
A good multi-model architecture does not promise that a provider failure will be invisible. It promises something more realistic: the failure will be bounded, the state will be preserved when possible, the fallback will be capability-aware, and the system will know when to stop pretending that another retry is recovery.
Related reading in the production AI series
For the model-selection layer, read Model Router for AI Agents. For replay-safe side effects, read Idempotent AI Actions. For stateful retries, see Durable Execution for AI Agents. For evaluation and trace design, continue with Agent Evals and Agent Observability.
Comments (0)