Niches · Model APIs & inference

Many small latency-sensitive calls on fast hardware

Two or three companies' docs show a full fit.

3companies, full fit
8companies, fit with a limit
58companies carry the tag
4share of the category's work
1first-hand accounts

What counts as done

The plan's published rate limits allow 100 calls of a 500-token answer in a row without a 429.

Measured, for our own tests: time to first token under 300 ms and output above 500 tokens a second on a 500-token answer, sustained for 100 calls without a 429. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: route or classify each message in under a second, I need 1,000 tokens a second for the voice agent, run gpt-oss on the fastest host

Does the job 3

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
Cerebras Inference B
Cerebras Systems Inc.
acc-12 measured Cerebras fastest for gpt-oss across six hosts, and the Hugging Face router lists 1,281 tokens a second at 182 ms first token on GPT OSS 120B; Developer tier has no hourly or daily caps. source
GroqCloud BB
Groq LLC
Hugging Face router lists gpt-oss-20b on Groq at 713 tokens a second with 288 ms first token and gpt-oss-120b at 426; acc-12 placed Groq second to Cerebras. Developer plan limits cover 100 calls; the free plan's 30 a minute does not. source
Hugging Face Inference Providers C
Hugging Face, Inc.
The router publishes throughput and first-token latency per provider and :fastest routes to the highest throughput; the list shows GPT OSS 120B on Cerebras at 1,281 tokens a second and 182 ms first token. source

Does the job, with a limit 8

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Cloudflare Workers AI BB
Cloudflare, Inc.
A -fast variant of Llama 3.3 70B exists, but no time-to-first-token or tokens-per-second figure is published and text generation is capped at 300 requests a minute. source
Fireworks AI Inference BB
Fireworks.ai, Inc.
Fast and Ultra-fast (Preview) variants exist with their own lower limits, but no time-to-first-token or tokens-per-second figure is published and Fireworks is absent from the Hugging Face router's throughput data. source
Novita AI C
Novita AI
Serves gpt-oss and other small models, but acc-12 measured Novita's throughput at about a seventh of Cerebras's on OpenRouter data, and no speed figures are published. source
OpenRouter B
OpenRouter, Inc.
sort throughput or latency and provider pinning pick fast hosts, but the same model id varies by host (acc-04 scored one host 20 points lower) and acc-12's comparison shows a 7x throughput spread. source
Prism Inference C
Prism Technologies Inc
Reasoning can be turned off to cut latency, but no time-to-first-token or throughput figure is published and per-key rate limits are unpublished. source
SambaCloud B
SambaNova Systems, Inc.
Served on SambaNova's RDUs, but no time-to-first-token or tokens-a-second figures in the docs read, and 60 RPM on gpt-oss-120b (Developer tier) is below 100 quick calls. source
Together AI Inference B
Together Computer, Inc.
Serverless targets TPS and TTFT goals but is best effort with 429 or 503 under demand; acc-12 ranks Together's gpt-oss throughput behind Cerebras; no figures published. source
Vercel AI Gateway B
Vercel Inc.
-fast model slugs route to a faster serving path at higher cost with fallback to the base model; no time-to-first-token or token-a-second figures and no published rate limits. source

Couldn't check 4

The docs were closed to us, or silent. These carry no mark either way.

ListingWhy
OpenAI API AA
OpenAI OpCo, LLC
Fast and Ultrafast modes exist per the pricing notes, but no latency or throughput figures were in the pages read.
Simplismart F
Verute Technologies Private Limited
No latency figures or rate limits in the docs read.
StepFun API E
STEPFUN TECHNOLOGY (HK) LIMITED
The host platform.stepfun.ai closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read.
Tencent Cloud TokenHub C
Tencent Cloud Computing (Beijing) Co., Ltd. (腾讯云计算(北京)有限责任公司)
The host cloud.tencent.com closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read.

Doesn't do this job 26

Listings in the category that do something else, with the reason.

ListingWhy
Amazon Bedrock A
Amazon Web Services, Inc.
Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published.
Claude API A
Anthropic, PBC
Not an open-weight fast host; fast mode on Opus 5.5 is a 2.5x throughput premium option with no published time to first token or tokens-per-second figure.
Antseed C
Antseed Foundation
Speed depends on the chosen seller; no latency or throughput figures are published.
Azure OpenAI in Foundry Models A
Microsoft Corporation
Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published.
BlockRun.AI A
BlockRun, Inc.
Not a fast host; a Nemotron Bolt model is described as about a second with no time-to-first-token or tokens-per-second figure.
Cloudflare AI Gateway B
Cloudflare, Inc.
A proxy adds a hop and publishes no latency figures.
Cohere Chat API (Command models) B
Cohere Inc.
Not a fast host; no latency or throughput figures published.
DeepInfra B
Deep Infra Inc.
The same gpt-oss-120b weights run at 46 tokens a second here against 1,281 on Cerebras in the Hugging Face router's listing, far below the 500 the use case needs. source
DeepSeek API D
Hangzhou DeepSeek Artificial Intelligence Co., Ltd.
Not a fast host; 36 status incidents and 503 overloads were recorded in the listing, and no latency figures are published.
Eden AI C
Eden AI SAS
A gateway hop with no latency figures and a 10 requests a second cap.
Glimpse F
Elorian AI, Inc.
A visual reasoning model whose reasoning tokens bill as output; no latency or throughput figures.
Gemini Developer API B
Google LLC
Not an open-weight fast host; no time-to-first-token or tokens-per-second figures.
Jais 3 C
Mohamed bin Zayed University of Artificial Intelligence (Institute of Foundation Models), with Inception42 (a G42 company)
No published latency or throughput; speed is the owner's hardware.
LiteLLM B
Berrie AI Incorporated
A proxy hop with no latency figures.
Meta Model API D
Meta Platforms, Inc.
Reasoning models that always reason; no latency or throughput figures.
MiniMax API C
Nanonoble Pte. Ltd.
No latency or throughput figures, and M3.1 Flash cannot turn thinking off (400 on disabled or effort none), so every short call pays reasoning tokens. source
Mistral AI API B
Mistral AI (RCS Paris 952 418 325)
No latency or throughput figures; rate limits are per-organisation numbers shown only in the console. source
Muse Spark Open C
Meta Platforms, Inc.
816B mixture of experts with mandatory reasoning at xhigh by default; no time-to-first-token or throughput figures. source
Nemotron 3.5 Super VL C
NVIDIA Corporation
12B active parameters but no time-to-first-token or throughput figures, reasoning is on by default, and the trial rate varies with other users' load. source
Router.com (Ramp Router) C
Ramp Business Corporation
Flex-tier routing is on by default and trades latency for cost; no time-to-first-token or throughput figures. source
Beam 1.0 C
Reflection AI
Reasoning is mandatory and defaults to max; no time-to-first-token or throughput figures. source
RouterPlus D
EVO HQ Inc
No time-to-first-token or throughput figures; the Optimize model search is paused. source
SiliconFlow E
SiliconFlow Labs Pte. Ltd.
No speed figures; a hosted open-weight catalogue, not a fast host. source
SWE-2 Flash F
Cognition AI, Inc.
No API, so no way to call it per message. source
Vertex AI (Gemini Enterprise Agent Platform) AA
Google LLC
A cloud platform; no time-to-first-token or throughput commitment outside Provisioned Throughput. source
xAI API (Grok) BB
SpaceXSI (xAI)
No time-to-first-token or tokens-a-second figures; priority processing is priced in the pricing notes without speed numbers, and RPS is RPM divided by 48. source

6 listings in the category were added after this reading and aren't marked yet.

What stops agents finishing

  • tokens-per-minute caps on the free and first paid tier below what an agent loop sends
  • fast hosts serve a short model list and retire models with little notice
  • same model id served at different speed and quality by different hosts

Around it

Before: speech.stt. After: speech.tts, inference.tool-loop.

Where this comes from

Weight 4 of 100 in Model APIs & inference: 1 first-hand comparison of 30 prompts x 5 runs plus provider benchmark tables; no Reddit or HN thread reached

The same as data: /api/v1/use-cases/inference.fast-small-calls.json, or the MCP tool get_use_case with id inference.fast-small-calls. Other jobs in this category: /api/v1/niches?category=inference.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.