Niches · Model APIs & inference
Keep an agent running through provider errors, rate limits and outages
Two or three companies' docs show a full fit.
What counts as done
A failed primary call falls through to the next model inside one request or one harness retry, the response says which provider served it, and the agent run finishes without the raw error reaching the user.
How agents ask for it: if the model is rate limited, try another one
, one key for several providers with fallback
, don't let a 429 or overloaded_error kill the cron job
Does the job 2
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| Eden AI C Eden AI SAS | A fallbacks list of up to 7 models retries inside one request on provider outage, rate limit or error, routing.allow_fallbacks switches provider for the same model, and provider/model ids pin a host. source |
| LiteLLM B Berrie AI Incorporated | fallbacks, context_window_fallbacks and content_policy_fallbacks run after num_retries with cooldowns, x-litellm-model-id names the deployment that answered, and acc-10 routes by failure class through it; a cross-provider fallback cannot decrypt the failed model's reasoning items. source |
Does the job, with a limit 10
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Antseed C Antseed Foundation | Automatic routing fails over after retryable errors and a `pick: failover` option exists, but a pinned `<peerId>@<model>` has no failover, 402 insufficient_deposits stops the run, and a model with no free seller fails with 502. source |
| BlockRun.AI A BlockRun, Inc. | The gateway fails over by itself and marks it with X-Fallback-Used, X-Original-Model and X-Fallback-Model on streaming and non-streaming responses, but the caller cannot set a model chain and the triggering errors are not listed; 503 says retry or fail over. source |
| Cloudflare AI Gateway B Cloudflare, Inc. | Universal endpoint fallbacks fire on any request error or timeout and cf-aig-step says which provider answered, and dynamic routes add rate and budget fallbacks; but the Universal endpoint is deprecated, dynamic routes need authentication plus stored BYOK keys, and mid-stream behaviour is not stated. source |
| DeepInfra B Deep Infra Inc. | No fallback list on the chat page read; fail_fast true turns a capacity queue into an immediate 429 so a client can fail over elsewhere, and service_tier in the response says which tier served. source |
| Glimpse F Elorian AI, Inc. | OpenRouter's models array falls back to another model on any error, but Glimpse has one endpoint, so provider failover has nowhere to go and a fallback means a different model. source |
| Hugging Face Inference Providers C Hugging Face, Inc. | :fastest, :cheapest and :preferred policies choose a live provider per request, but nothing read says a failed call falls through to another provider inside one request, and no 429 or error format is documented for the router. source |
| OpenRouter B OpenRouter, Inc. | models (or fallbacks, max 3) retries on any error including rate limits and moderation, and the chunk names the provider; once headers are committed errors arrive inside a 200 stream, and acc-01, acc-02 and acc-21 report failover not firing or shape differences. source |
| Router.com (Ramp Router) C Ramp Business Corporation | models (up to 15) on /v1/responses advances on unknown or retired model, 429, 5xx, network failure or pre-stream failure and names the provider; it stops once streaming starts, and truncated attempts are billed. source |
| RouterPlus D EVO HQ Inc | Fails over between providers of one model before output starts, with x-tm-attempts and x-tm-provider naming the server; it never fails over to another model, and after output starts a failure ends the stream. source |
| Vercel AI Gateway B Vercel Inc. | Default routing fails over across a model's providers and a models array tries backups in order, with modelAttempts and resolvedProvider naming what served; whether fallback continues after streaming starts isn't stated. source |
Couldn't check 2
The docs were closed to us, or silent. These carry no mark either way.
| Listing | Why |
|---|---|
| StepFun API E STEPFUN TECHNOLOGY (HK) LIMITED | The host platform.stepfun.ai closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read. |
| Tencent Cloud TokenHub C Tencent Cloud Computing (Beijing) Co., Ltd. (腾讯云计算(北京)有限责任公司) | The host cloud.tencent.com closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read. |
Doesn't do this job 27
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Amazon Bedrock A Amazon Web Services, Inc. | No fallback or models parameter; cross-Region inference profiles spread capacity for one model and do not fall through to another model. |
| Claude API A Anthropic, PBC | Single-provider API with no models array or fallback parameter; a spend-cap 429 carries no retry-after and SDK retries fail until the month resets. |
| Azure OpenAI in Foundry Models A Microsoft Corporation | No model fallback parameter; spillover only routes a provisioned deployment's overflow to a standard deployment of the same model in the same resource. source |
| Cerebras Inference B Cerebras Systems Inc. | Single host with no fallback or models parameter. |
| Cloudflare Workers AI BB Cloudflare, Inc. | No fallback parameter; failover is AI Gateway's job. |
| Cohere Chat API (Command models) B Cohere Inc. | Single-vendor API with no fallback parameter. |
| DeepSeek API D Hangzhou DeepSeek Artificial Intelligence Co., Ltd. | Single-vendor API with no fallback parameter; 429 and 503 carry no Retry-After. |
| Fireworks AI Inference BB Fireworks.ai, Inc. | Single host with no fallback or models parameter. |
| Gemini Developer API B Google LLC | Single-vendor API with no fallback parameter; the Gemini CLI's Pro-to-Flash switch in acc-16 was inconsistent and lives in the harness. |
| GroqCloud BB Groq LLC | Single host with no fallback or models parameter; 498 capacity_exceeded on Flex is a retry signal, not a fallback. |
| Jais 3 C Mohamed bin Zayed University of Artificial Intelligence (Institute of Foundation Models), with Inception42 (a G42 company) | Self-hosted weights; no router. |
| Meta Model API D Meta Platforms, Inc. | Single-vendor API with no fallback parameter; service_overloaded 429s carry Retry-After 60 and are retryable. |
| MiniMax API C Nanonoble Pte. Ltd. | Single-vendor API with no models list or fallback parameter; 429, 529 and the 1002 and 2064 retry-later codes are returned for the caller to handle. source |
| Mistral AI API B Mistral AI (RCS Paris 952 418 325) | Single vendor; the error glossary gives Retry-After on 429 and backoff advice, with no models list or fallback parameter. source |
| Muse Spark Open C Meta Platforms, Inc. | Self-hosted weights; no router, models list or fallback. source |
| Nemotron 3.5 Super VL C NVIDIA Corporation | Single hosted endpoint with no fallback; the FAQ says request rates vary with other users and gives no 429 contract. source |
| Novita AI C Novita AI | Single provider; errors come back as codes such as RATE_LIMIT_EXCEEDED with no fallback parameter. source |
| OpenAI API AA OpenAI OpCo, LLC | No fallback parameter; 429 slow_down and 503 server_is_overloaded carry Retry-After, and acc-03 and acc-10 describe rate-limit and billing failures killing agent runs until a harness-side fallback was added. source |
| Prism Inference C Prism Technologies Inc | Single provider; a 429 covers both key limits and model capacity with Retry-After and a retryable flag, and there is no fallback list. source |
| Beam 1.0 C Reflection AI | Single provider; 429 and 503 carry Retry-After and x-should-retry, with no fallback list. source |
| SambaCloud B SambaNova Systems, Inc. | Single provider with headers for remaining requests; no fallback list. source |
| SiliconFlow E SiliconFlow Labs Pte. Ltd. | Single provider; 429 messages name the limit type and 503 or 504 mean high load, with no fallback list. source |
| Simplismart F Verute Technologies Private Limited | Single provider; no fallback, 429 format or retry guidance documented. source |
| SWE-2 Flash F Cognition AI, Inc. | No API; model choice is a CLI flag or slash command. source |
| Together AI Inference B Together Computer, Inc. | Single provider; 429 and 503 mean high demand with backoff advice and no fallback list. source |
| Vertex AI (Gemini Enterprise Agent Platform) AA Google LLC | No fallback inside the platform; a 429 'Resource exhausted' on Standard PayGo means shared capacity, and the guide's remedies are backoff, the global endpoint or Provisioned Throughput. source |
| xAI API (Grok) BB SpaceXSI (xAI) | Single provider; 429 is 'sending requests too frequently' with no Retry-After documented and no fallback list. source |
6 listings in the category were added after this reading and aren't marked yet.
What stops agents finishing
- router returns a generic error string the harness cannot classify, so no fallback is tried
- fallback stops once a stream has started
- spend cap or billing failure returns an error the harness retries instead of stopping
- free-tier models all return 429 under a burst, so a chain of free models is no chain
Around it
Before: inference.tool-loop. After: obs.gateway, obs.cost-tracking.
Where this comes from
Weight 16 of 100 in Model APIs & inference: 8 of 28 accounts are about fallback that did or did not fire; OpenClaw's issue tracker lists broken failover as the top OpenRouter complaint
The same as data: /api/v1/use-cases/inference.failover-routing.json, or the MCP tool get_use_case with id inference.failover-routing. Other jobs in this category: /api/v1/niches?category=inference.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.