# Many small latency-sensitive calls on fast hardware

> Many small latency-sensitive calls on fast hardware: 3 companies fully do it, 8 with a limit, 58 carry the tag. Every listing in Model APIs & inference with the reason and a source.

- Canonical: https://www.anchorterminal.com/niches/inference.fast-small-calls/
- Markdown: https://www.anchorterminal.com/niches/inference.fast-small-calls/index.md (~2,450 tokens)
- Slim: https://www.anchorterminal.com/niches/inference.fast-small-calls/index.min.md (~380 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/niches/inference.fast-small-calls/index.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-15

Two or three companies' docs show a full fit.

| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |
| --- | --- | --- | --- | --- |
| 3 | 8 | 58 | 4 | 1 |

## What counts as done

The plan's published rate limits allow 100 calls of a 500-token answer in a row without a 429.

Measured, for our own tests: time to first token under 300 ms and output above 500 tokens a second on a 500-token answer, sustained for 100 calls without a 429.

How agents ask for it: route or classify each message in under a second; I need 1,000 tokens a second for the voice agent; run gpt-oss on the fastest host

## Does the job

The docs show it meets the test above, and nothing we read says otherwise.

- **Cerebras Inference (Cerebras Systems Inc.)**: acc-12 measured Cerebras fastest for gpt-oss across six hosts, and the Hugging Face router lists 1,281 tokens a second at 182 ms first token on GPT OSS 120B; Developer tier has no hourly or daily caps.
- **GroqCloud (Groq LLC)**: Hugging Face router lists gpt-oss-20b on Groq at 713 tokens a second with 288 ms first token and gpt-oss-120b at 426; acc-12 placed Groq second to Cerebras. Developer plan limits cover 100 calls; the free plan's 30 a minute does not.
- **Hugging Face Inference Providers (Hugging Face, Inc.)**: The router publishes throughput and first-token latency per provider and :fastest routes to the highest throughput; the list shows GPT OSS 120B on Cerebras at 1,281 tokens a second and 182 ms first token.

## Does the job, with a limit

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

- **Cloudflare Workers AI (Cloudflare, Inc.)**: A -fast variant of Llama 3.3 70B exists, but no time-to-first-token or tokens-per-second figure is published and text generation is capped at 300 requests a minute.
- **Fireworks AI Inference (Fireworks.ai, Inc.)**: Fast and Ultra-fast (Preview) variants exist with their own lower limits, but no time-to-first-token or tokens-per-second figure is published and Fireworks is absent from the Hugging Face router's throughput data.
- **Novita AI (Novita AI)**: Serves gpt-oss and other small models, but acc-12 measured Novita's throughput at about a seventh of Cerebras's on OpenRouter data, and no speed figures are published.
- **OpenRouter (OpenRouter, Inc.)**: sort throughput or latency and provider pinning pick fast hosts, but the same model id varies by host (acc-04 scored one host 20 points lower) and acc-12's comparison shows a 7x throughput spread.
- **Prism Inference (Prism Technologies Inc)**: Reasoning can be turned off to cut latency, but no time-to-first-token or throughput figure is published and per-key rate limits are unpublished.
- **SambaCloud (SambaNova Systems, Inc.)**: Served on SambaNova's RDUs, but no time-to-first-token or tokens-a-second figures in the docs read, and 60 RPM on gpt-oss-120b (Developer tier) is below 100 quick calls.
- **Together AI Inference (Together Computer, Inc.)**: Serverless targets TPS and TTFT goals but is best effort with 429 or 503 under demand; acc-12 ranks Together's gpt-oss throughput behind Cerebras; no figures published.
- **Vercel AI Gateway (Vercel Inc.)**: -fast model slugs route to a faster serving path at higher cost with fallback to the base model; no time-to-first-token or token-a-second figures and no published rate limits.

## Couldn't check

The docs were closed to us, or silent. These carry no mark either way.

- **OpenAI API (OpenAI OpCo, LLC)**: Fast and Ultrafast modes exist per the pricing notes, but no latency or throughput figures were in the pages read.
- **Simplismart (Verute Technologies Private Limited)**: No latency figures or rate limits in the docs read.
- **StepFun API (STEPFUN TECHNOLOGY (HK) LIMITED)**: The host platform.stepfun.ai closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read.
- **Tencent Cloud TokenHub (Tencent Cloud Computing (Beijing) Co., Ltd. (腾讯云计算（北京）有限责任公司))**: The host cloud.tencent.com closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read.

- not this job: Amazon Bedrock (Amazon Web Services, Inc.): Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published.
- not this job: Claude API (Anthropic, PBC): Not an open-weight fast host; fast mode on Opus 5.5 is a 2.5x throughput premium option with no published time to first token or tokens-per-second figure.
- not this job: Antseed (Antseed Foundation): Speed depends on the chosen seller; no latency or throughput figures are published.
- not this job: Azure OpenAI in Foundry Models (Microsoft Corporation): Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published.
- not this job: BlockRun.AI (BlockRun, Inc.): Not a fast host; a Nemotron Bolt model is described as about a second with no time-to-first-token or tokens-per-second figure.
- not this job: Cloudflare AI Gateway (Cloudflare, Inc.): A proxy adds a hop and publishes no latency figures.
- not this job: Cohere Chat API (Command models) (Cohere Inc.): Not a fast host; no latency or throughput figures published.
- not this job: DeepInfra (Deep Infra Inc.): The same gpt-oss-120b weights run at 46 tokens a second here against 1,281 on Cerebras in the Hugging Face router's listing, far below the 500 the use case needs.
- not this job: DeepSeek API (Hangzhou DeepSeek Artificial Intelligence Co., Ltd.): Not a fast host; 36 status incidents and 503 overloads were recorded in the listing, and no latency figures are published.
- not this job: Eden AI (Eden AI SAS): A gateway hop with no latency figures and a 10 requests a second cap.
- not this job: Glimpse (Elorian AI, Inc.): A visual reasoning model whose reasoning tokens bill as output; no latency or throughput figures.
- not this job: Gemini Developer API (Google LLC): Not an open-weight fast host; no time-to-first-token or tokens-per-second figures.
- not this job: Jais 3 (Mohamed bin Zayed University of Artificial Intelligence (Institute of Foundation Models), with Inception42 (a G42 company)): No published latency or throughput; speed is the owner's hardware.
- not this job: LiteLLM (Berrie AI Incorporated): A proxy hop with no latency figures.
- not this job: Meta Model API (Meta Platforms, Inc.): Reasoning models that always reason; no latency or throughput figures.
- not this job: MiniMax API (Nanonoble Pte. Ltd.): No latency or throughput figures, and M3.1 Flash cannot turn thinking off (400 on disabled or effort none), so every short call pays reasoning tokens.
- not this job: Mistral AI API (Mistral AI (RCS Paris 952 418 325)): No latency or throughput figures; rate limits are per-organisation numbers shown only in the console.
- not this job: Muse Spark Open (Meta Platforms, Inc.): 816B mixture of experts with mandatory reasoning at xhigh by default; no time-to-first-token or throughput figures.
- not this job: Nemotron 3.5 Super VL (NVIDIA Corporation): 12B active parameters but no time-to-first-token or throughput figures, reasoning is on by default, and the trial rate varies with other users' load.
- not this job: Router.com (Ramp Router) (Ramp Business Corporation): Flex-tier routing is on by default and trades latency for cost; no time-to-first-token or throughput figures.
- not this job: Beam 1.0 (Reflection AI): Reasoning is mandatory and defaults to max; no time-to-first-token or throughput figures.
- not this job: RouterPlus (EVO HQ Inc): No time-to-first-token or throughput figures; the Optimize model search is paused.
- not this job: SiliconFlow (SiliconFlow Labs Pte. Ltd.): No speed figures; a hosted open-weight catalogue, not a fast host.
- not this job: SWE-2 Flash (Cognition AI, Inc.): No API, so no way to call it per message.
- not this job: Vertex AI (Gemini Enterprise Agent Platform) (Google LLC): A cloud platform; no time-to-first-token or throughput commitment outside Provisioned Throughput.
- not this job: xAI API (Grok) (SpaceXSI (xAI)): No time-to-first-token or tokens-a-second figures; priority processing is priced in the pricing notes without speed numbers, and RPS is RPM divided by 48.

6 listings in the category were added after this reading and aren't marked yet.

## What stops agents finishing

- tokens-per-minute caps on the free and first paid tier below what an agent loop sends
- fast hosts serve a short model list and retire models with little notice
- same model id served at different speed and quality by different hosts

## Around it

Before: speech.stt. After: speech.tts, inference.tool-loop.

## Where this comes from

Weight 4 of 100 in Model APIs & inference: 1 first-hand comparison of 30 prompts x 5 runs plus provider benchmark tables; no Reddit or HN thread reached

As data: https://www.anchorterminal.com/api/v1/use-cases/inference.fast-small-calls.json, or the MCP tool get_use_case with id inference.fast-small-calls. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
