Guides2026-10-167 min read

LLM API Fallback: Keep Your App Up When a Model Fails

An LLM API fallback plan decides what your application does when a model call fails: wait and retry, send the same request somewhere else, or ask a different model. Without one, a provider's bad hour becomes your outage. With a careless one, a retry loop can double your bill or send a user two half-answers. This guide covers the failures that happen in practice, the patterns that handle them, a short Python implementation, and which parts a multi-route API already takes off your hands.

What actually fails in production

LLM calls fail in a handful of distinct ways, and each needs a different response.

Rate limits (429). Too many requests or tokens per minute for an account, a key or a model. Usually temporary, and the response often says how long to wait.
Server errors (500, 502, 503). The model's service is overloaded or broken. Often temporary, sometimes for hours.
Timeouts (504 or a client timeout). No answer within the time allowed. A timeout is not always an outage: when we sent a long coding prompt to two reasoning models on 16 October 2026 without streaming, neither had finished within the 180 seconds limit for non-streaming requests, and both returned provider_timeout.
Broken streams. The connection drops after part of the answer has arrived.
Request errors (400). The request itself is wrong: an unknown model, a parameter the model does not accept, input longer than its context window. Retrying the same request gives the same error.
Account errors (401, 402, 403). A bad key, an empty balance, a disabled key. Retrying makes no difference until someone fixes the account.

Retry, failover and fallback are different things

The three words get used loosely. Keeping them apart makes the design easier.

Retry sends the same request to the same place again after a pause. It fixes brief overloads and rate limits.
Failover sends the same request for the same model to another place that serves it: another region, deployment or provider. The answer should be equivalent, because the model is the same.
Fallback sends the request to a different model. It keeps the application answering when a model is unavailable everywhere, at the price of a different answer: other strengths, other formatting, sometimes other tool-calling habits.

Retry and failover are safe to automate. Fallback to another model is a product decision, because users may notice the change.

LLM API fallback patterns that work

1. Classify the error before acting. A small table in code prevents the two classic mistakes: retrying requests that can never succeed, and giving up on errors that would clear in seconds.

Error	Retry	Fail over	Fall back to another model
429 rate limit	Yes, after the wait it asks for	Yes	If it persists
500, 502, 503	Yes, with backoff	Yes	If it persists
504 or client timeout	Stream instead, or raise the limit	Yes	If it persists
400 bad request	No; fix the request	No	Only if another model accepts the request
401, 402, 403	No; fix the account	No	No

2. Back off exponentially, with jitter, and respect retry-after. Doubling the wait between attempts and adding a random offset stops many clients from retrying in lockstep. When the response carries retry-after, wait at least that long.

3. Stream long generations and set timeouts in layers. A connect timeout catches a dead network. A first-token timeout catches a stuck request. A total timeout stops a runaway answer. For prompts that make a model think for minutes, streaming keeps the connection alive and lets the client see progress, where a non-streaming request would hit a time limit first.

4. Keep the fallback chain short and tested. Two or three models, chosen in advance, each tested with your real prompts. A chain of models nobody has tried with your prompts is a guess, not a plan.

5. Stop sending traffic to a target that keeps failing. After repeated failures, pause requests to that model or endpoint for a short time instead of letting every request discover the same outage. This is the circuit breaker pattern, and it saves both latency and money during an incident.

6. Do not silently retry a stream the user is already reading. If text has reached the user and the stream breaks, a fresh request produces a different answer, not a continuation. Show what arrived, mark it as incomplete, and offer a retry.

7. Log the request id. Every failed call should record the model, the error code and the request id. That turns "the AI was down this morning" into something support can trace.

A minimal implementation in Python

The OpenAI Python SDK already retries connection errors, 408, 409, 429 and 5xx responses with exponential backoff, twice by default, as its README describes. The code below adds the part it does not do: moving to the next model in a short chain when one stays unavailable, while letting request and account errors surface immediately.

import openai
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.boostrail.com/v1",
    max_retries=2,
    timeout=120,
)

FALLBACK_CHAIN = ["claude-sonnet-5", "kimi-k3", "deepseek-v4.1-flash"]
RETRYABLE = (
    openai.RateLimitError,
    openai.APITimeoutError,
    openai.APIConnectionError,
    openai.InternalServerError,
)


def complete(messages):
    last_error = None
    for model in FALLBACK_CHAIN:
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except RETRYABLE as error:
            last_error = error
    raise last_error


reply = complete([{"role": "user", "content": "Summarise this incident report in three bullet points."}])
print(reply.model, reply.choices[0].message.content)

Because every model sits behind the same endpoint and key, the chain is a list of strings. reply.model tells you which model answered, which is worth logging and, in some products, showing.

What a multi-route API handles for you

On BoostRail, failover for the same model happens before the response reaches you. The rules are published on Errors and failover:

When the route serving a request fails, BoostRail tries the same model on another route before anything is sent to you. It never switches to a different model.
Server errors, rate limiting and the route refusing BoostRail's own access are moved to another route. Request errors and non-streaming timeouts are not, because another route would fail the same way.
For streaming requests, if no first event arrives within 45 seconds, the request moves to the next route.
Once any data has been sent to you, the request is not moved.
When every route fails, you get the error from the last attempt, and the request is not charged.

That changes what your own code needs to do. Failover for the same model is covered, so your retries mostly handle what is left: 429 with retry-after, and 502 or 504 after BoostRail has already tried its routes. Fallback to a different model stays your decision, because BoostRail does not make it for you. The price is the model's published price on the Models page, whichever route serves the request.

For the broader integration problems that sit around fallback, such as schema differences and key sprawl, see The Hidden Complexity of Multi-API AI Integrations.

Testing the fallback path

A fallback path that has never run will fail the first time it matters. Three cheap ways to exercise it:

Force a timeout. Give the first model in the chain a client timeout far shorter than any real answer, and check that the second model answers and the log records why.
Break the network. Point the client at an address that refuses connections and confirm the error is classified as retryable, not as a request error.
Check the account errors. Use a disabled key in a test environment and confirm the application reports the problem instead of cycling through the chain.

Run these whenever the chain changes, and watch the cost of a test run: every attempt that succeeds is billed.

Frequently asked questions

Should my application switch to a different model automatically?

Only to models you have tested with your real prompts, and only when users can accept a different answer style. For internal tools and batch jobs, automatic fallback is usually fine. For customer-facing features, log or show which model answered.

How long should LLM API timeouts be?

Long enough for the slowest legitimate answer, and no longer. For anything that may reason or write at length, stream the response: on BoostRail a non-streaming request is limited to 180 seconds, while a stream only needs its first event within 45 seconds on each route.

Am I charged for failed attempts?

On BoostRail, a request that fails on every route is not charged. Each retry or fallback your code sends is a new request, billed normally if it succeeds.

What happens if a stream breaks halfway?

BoostRail does not move a request once data has been sent, because a new attempt would produce a different answer. The client should keep what arrived, mark it incomplete and offer a retry. Billing for interrupted streams follows the rule on the pricing page.

Is retrying a 400 error ever useful?

Only after changing the request: trimming the input to fit the context window, removing a parameter the model rejects, or choosing a model that accepts it. Sending the same request again returns the same error.