Niches · Model APIs & inference
Many small latency-sensitive calls on fast hardware
Two or three companies' docs show a full fit.
What counts as done
The plan's published rate limits allow 100 calls of a 500-token answer in a row without a 429.
Measured, for our own tests: time to first token under 300 ms and output above 500 tokens a second on a 500-token answer, sustained for 100 calls without a 429. No docs we read state these figures, so they don't count against a listing here.
How agents ask for it: route or classify each message in under a second
, I need 1,000 tokens a second for the voice agent
, run gpt-oss on the fastest host
Does the job 3
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| Cerebras Inference B Cerebras Systems Inc. | acc-12 measured Cerebras fastest for gpt-oss across six hosts, and the Hugging Face router lists 1,281 tokens a second at 182 ms first token on GPT OSS 120B; Developer tier has no hourly or daily caps. source |
| GroqCloud BB Groq LLC | Hugging Face router lists gpt-oss-20b on Groq at 713 tokens a second with 288 ms first token and gpt-oss-120b at 426; acc-12 placed Groq second to Cerebras. Developer plan limits cover 100 calls; the free plan's 30 a minute does not. source |
| Hugging Face Inference Providers C Hugging Face, Inc. | The router publishes throughput and first-token latency per provider and :fastest routes to the highest throughput; the list shows GPT OSS 120B on Cerebras at 1,281 tokens a second and 182 ms first token. source |
Does the job, with a limit 8
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Cloudflare Workers AI BB Cloudflare, Inc. | A -fast variant of Llama 3.3 70B exists, but no time-to-first-token or tokens-per-second figure is published and text generation is capped at 300 requests a minute. source |
| Fireworks AI Inference BB Fireworks.ai, Inc. | Fast and Ultra-fast (Preview) variants exist with their own lower limits, but no time-to-first-token or tokens-per-second figure is published and Fireworks is absent from the Hugging Face router's throughput data. source |
| Novita AI C Novita AI | Serves gpt-oss and other small models, but acc-12 measured Novita's throughput at about a seventh of Cerebras's on OpenRouter data, and no speed figures are published. source |
| OpenRouter B OpenRouter, Inc. | sort throughput or latency and provider pinning pick fast hosts, but the same model id varies by host (acc-04 scored one host 20 points lower) and acc-12's comparison shows a 7x throughput spread. source |
| Prism Inference C Prism Technologies Inc | Reasoning can be turned off to cut latency, but no time-to-first-token or throughput figure is published and per-key rate limits are unpublished. source |
| SambaCloud B SambaNova Systems, Inc. | Served on SambaNova's RDUs, but no time-to-first-token or tokens-a-second figures in the docs read, and 60 RPM on gpt-oss-120b (Developer tier) is below 100 quick calls. source |
| Together AI Inference B Together Computer, Inc. | Serverless targets TPS and TTFT goals but is best effort with 429 or 503 under demand; acc-12 ranks Together's gpt-oss throughput behind Cerebras; no figures published. source |
| Vercel AI Gateway B Vercel Inc. | -fast model slugs route to a faster serving path at higher cost with fallback to the base model; no time-to-first-token or token-a-second figures and no published rate limits. source |
Couldn't check 4
The docs were closed to us, or silent. These carry no mark either way.
| Listing | Why |
|---|---|
| OpenAI API AA OpenAI OpCo, LLC | Fast and Ultrafast modes exist per the pricing notes, but no latency or throughput figures were in the pages read. |
| Simplismart F Verute Technologies Private Limited | No latency figures or rate limits in the docs read. |
| StepFun API E STEPFUN TECHNOLOGY (HK) LIMITED | The host platform.stepfun.ai closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read. |
| Tencent Cloud TokenHub C Tencent Cloud Computing (Beijing) Co., Ltd. (腾讯云计算(北京)有限责任公司) | The host cloud.tencent.com closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read. |
Doesn't do this job 26
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Amazon Bedrock A Amazon Web Services, Inc. | Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published. |
| Claude API A Anthropic, PBC | Not an open-weight fast host; fast mode on Opus 5.5 is a 2.5x throughput premium option with no published time to first token or tokens-per-second figure. |
| Antseed C Antseed Foundation | Speed depends on the chosen seller; no latency or throughput figures are published. |
| Azure OpenAI in Foundry Models A Microsoft Corporation | Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published. |
| BlockRun.AI A BlockRun, Inc. | Not a fast host; a Nemotron Bolt model is described as about a second with no time-to-first-token or tokens-per-second figure. |
| Cloudflare AI Gateway B Cloudflare, Inc. | A proxy adds a hop and publishes no latency figures. |
| Cohere Chat API (Command models) B Cohere Inc. | Not a fast host; no latency or throughput figures published. |
| DeepInfra B Deep Infra Inc. | The same gpt-oss-120b weights run at 46 tokens a second here against 1,281 on Cerebras in the Hugging Face router's listing, far below the 500 the use case needs. source |
| DeepSeek API D Hangzhou DeepSeek Artificial Intelligence Co., Ltd. | Not a fast host; 36 status incidents and 503 overloads were recorded in the listing, and no latency figures are published. |
| Eden AI C Eden AI SAS | A gateway hop with no latency figures and a 10 requests a second cap. |
| Glimpse F Elorian AI, Inc. | A visual reasoning model whose reasoning tokens bill as output; no latency or throughput figures. |
| Gemini Developer API B Google LLC | Not an open-weight fast host; no time-to-first-token or tokens-per-second figures. |
| Jais 3 C Mohamed bin Zayed University of Artificial Intelligence (Institute of Foundation Models), with Inception42 (a G42 company) | No published latency or throughput; speed is the owner's hardware. |
| LiteLLM B Berrie AI Incorporated | A proxy hop with no latency figures. |
| Meta Model API D Meta Platforms, Inc. | Reasoning models that always reason; no latency or throughput figures. |
| MiniMax API C Nanonoble Pte. Ltd. | No latency or throughput figures, and M3.1 Flash cannot turn thinking off (400 on disabled or effort none), so every short call pays reasoning tokens. source |
| Mistral AI API B Mistral AI (RCS Paris 952 418 325) | No latency or throughput figures; rate limits are per-organisation numbers shown only in the console. source |
| Muse Spark Open C Meta Platforms, Inc. | 816B mixture of experts with mandatory reasoning at xhigh by default; no time-to-first-token or throughput figures. source |
| Nemotron 3.5 Super VL C NVIDIA Corporation | 12B active parameters but no time-to-first-token or throughput figures, reasoning is on by default, and the trial rate varies with other users' load. source |
| Router.com (Ramp Router) C Ramp Business Corporation | Flex-tier routing is on by default and trades latency for cost; no time-to-first-token or throughput figures. source |
| Beam 1.0 C Reflection AI | Reasoning is mandatory and defaults to max; no time-to-first-token or throughput figures. source |
| RouterPlus D EVO HQ Inc | No time-to-first-token or throughput figures; the Optimize model search is paused. source |
| SiliconFlow E SiliconFlow Labs Pte. Ltd. | No speed figures; a hosted open-weight catalogue, not a fast host. source |
| SWE-2 Flash F Cognition AI, Inc. | No API, so no way to call it per message. source |
| Vertex AI (Gemini Enterprise Agent Platform) AA Google LLC | A cloud platform; no time-to-first-token or throughput commitment outside Provisioned Throughput. source |
| xAI API (Grok) BB SpaceXSI (xAI) | No time-to-first-token or tokens-a-second figures; priority processing is priced in the pricing notes without speed numbers, and RPS is RPM divided by 48. source |
6 listings in the category were added after this reading and aren't marked yet.
What stops agents finishing
- tokens-per-minute caps on the free and first paid tier below what an agent loop sends
- fast hosts serve a short model list and retire models with little notice
- same model id served at different speed and quality by different hosts
Around it
Before: speech.stt. After: speech.tts, inference.tool-loop.
Where this comes from
Weight 4 of 100 in Model APIs & inference: 1 first-hand comparison of 30 prompts x 5 runs plus provider benchmark tables; no Reddit or HN thread reached
The same as data: /api/v1/use-cases/inference.fast-small-calls.json, or the MCP tool get_use_case with id inference.fast-small-calls. Other jobs in this category: /api/v1/niches?category=inference.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.