LLM Continuity

AI providers have outages, hit capacity, and run out of credit. Without a backup, every one of those becomes an outage for your agents too.

Continuity lets you add backup providers to an LLM connection. When a call to the main provider fails, the gateway tries the backups in order before the agent ever sees an error.

Quick Start: What do you want to do?

I want to...	Go to
Add a backup provider	Adding a backup
Choose which model the backup answers with	Choosing the model a backup uses
Change what counts as a failure	What triggers a move to a backup?
Stop one agent falling back	Narrowing it for one persona
Know which provider answered	How do I know which provider answered?
Understand what it costs	How are backups billed?

---

Adding a backup

1. Open the connection from the LLM Registry.
2. Go to Configuration → Safeguards → Continuity.
3. Select Add a backup and choose another of your LLM connections.
4. Choose the model it should answer with.
5. Add more backups if you want, and use the arrows to order them.

A connection with no backups shows Single point of failure — if that provider is down, every call through it fails. Once you add one, the banner changes to show how many backups protect it.

Backups are tried in order, top first. You can have up to three per connection.

The If a call fails, the gateway tries card shows the resulting chain — the primary, then each backup with the model it will answer with and what it costs — so you can read the whole arrangement at a glance.

Choosing the model a backup uses

A backup is a different provider, so it may not offer the same model names.

Option	What it does	Use it when
Same model	The agent keeps the model it asked for, under the backup's name for it	Almost always. Agents behave the same on the backup.
One model for everything	Every call reaching this backup uses one model you pick	A deliberate downgrade, or a different vendor entirely
Choose per model	You map each of the connection's models to a backup model	The naming does not line up and you want to decide each case

Below your choice, a coverage preview tells you exactly which models the backup can answer for. If some cannot be matched, it says so — calls for those models skip this backup and move to the next one rather than failing.

If a backup speaks a different provider's API, the gateway converts the call. Features that only exist on one provider, such as prompt caching or extended thinking, may not carry over, and context windows differ — a prompt that is too long for the backup fails rather than moving on again. The panel warns you when the backup differs in either way.

What triggers a move to a backup?

Five kinds of failure can trigger a move. All are on by default, and at least one must stay on.

Trigger	What it covers
Provider down or overloaded	Connection failures, 5xx and overload errors, or a stream that sends nothing for 30 seconds
Rate limited	Throttling responses. Turn this off to pass the limit back to the agent so it retries the same provider.
Out of credit or quota	The provider account has run out. That provider is skipped for 5 minutes.
The provider refuses the gateway's key	Only for keys the gateway holds. A key the agent supplies itself is the agent's problem to fix.
Model not available there	The model is retired, or not enabled for that account or region.

What never moves to a backup?

Some failures are deliberately not retried elsewhere:

• A provider's content filter. Retrying elsewhere would be a way around it.
• A problem with the request itself, which would fail on any provider.
• Your own controls — request controls, Data Governance, Prompt Guard, budgets and Model Registry blocks. Falling back would route around the control you configured.
• A response that has already started reaching the agent. Once output is flowing, the gateway does not restart it elsewhere.

A backup is also skipped — not failed over to — if it does not offer the model, is itself down, or is cooling off after a limit.

What happens when a provider does not answer?

A stuck provider looks exactly like a slow one: silence. Two settings decide how long the agent waits.

Setting	Behaviour
Streaming answers	Move on after 30 seconds of silence. Always on, not configurable.
Non-streaming answers	Wait as long as it takes by default, or choose a limit between 15 and 180 seconds.

Streaming providers normally start answering within a second or two, so 30 seconds of nothing reliably means stuck.

For non-streaming, keep any limit below your load balancer's idle timeout — often 60 seconds — so the agent gets a real answer rather than a gateway timeout. Note that an answer which legitimately needs longer than your limit will also be sent to a backup.

The last backup in the chain is always allowed to finish.

Narrowing it for one persona

By default a persona uses whatever its connections offer. You can change that per persona, under Configuration → LLMs, in the block headed If an LLM fails.

Option	Effect
Use the connection's backups	The default. Changes made on the connection apply here automatically.
Stay on provider	This persona never falls back. It gets the error instead. Use it when data must not leave a particular provider.
Only some backups or failures	Pick which of the connection's backups and triggers this persona may use.

A persona can only ever narrow what the connection allows — it can never add a backup or a trigger the connection does not have.

Two things to watch for:

• A persona can only fall back to LLMs it has been granted. Backups it has not been granted are skipped, and the panel offers to grant them. Granting gives the persona all of that connection's models, not just fallback use.
• If the connection's backups change afterwards, a persona using Only some backups or failures keeps its original selection and will warn you that it no longer matches.

How do I know which provider answered?

When a backup answers, the response carries headers naming the provider and model that served it. The same information appears in Agent Activity, so you can see after the fact how often you are running on a backup — a useful signal that a primary provider is becoming unreliable.

How are backups billed?

Each backup uses its own credentials and its own rates, and one provider's keys are never sent to another.

For budgets, the gateway holds the estimated cost at the primary's rates when the call starts, then settles at the rates of whichever provider actually answered. A backup that is more expensive therefore shows up correctly in Budgets and AI Spend.

Each backup also applies its own safeguards, so a call that moves to a backup is still subject to that connection's controls.

Real-world examples

Keep a customer-facing assistant online

Setting	Value
Primary	Your main provider
Backup 1	A second provider offering the same model
Model on backup	Same model
Triggers	All five

Agents keep working through an outage, with identical behaviour because the model is unchanged.

Protect a regulated workload from leaving its provider

Setting	Value
Connection	Has backups configured for general use
Persona	Stay on provider

Other agents fall back; this one fails instead, because an answer from elsewhere would be worse than no answer.

Tips

• Prefer a backup that serves the same model. It is the only option where agent behaviour is genuinely unchanged.
• Add the backup before you need it. The banner tells you when a connection is a single point of failure — treat it as a to-do.
• Check the coverage preview. A backup that cannot answer for half your models is doing less than you think.
• Leave "Rate limited" on unless your client retries well. Passing a limit back to an agent that retries immediately just moves the problem.
• Watch how often backups answer. Agent Activity shows it, and a rising count is early warning about your primary provider.