{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/inference.fast-small-calls/",
    "json": "https://www.anchorterminal.com/niches/inference.fast-small-calls/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/inference.fast-small-calls/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/inference.fast-small-calls/index.min.md"
  },
  "markdown": "Two or three companies' docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 3 | 8 | 58 | 4 | 1 |\n\n## What counts as done\n\nThe plan's published rate limits allow 100 calls of a 500-token answer in a row without a 429.\n\nMeasured, for our own tests: time to first token under 300 ms and output above 500 tokens a second on a 500-token answer, sustained for 100 calls without a 429.\n\nHow agents ask for it: route or classify each message in under a second; I need 1,000 tokens a second for the voice agent; run gpt-oss on the fastest host\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **Cerebras Inference (Cerebras Systems Inc.)**: acc-12 measured Cerebras fastest for gpt-oss across six hosts, and the Hugging Face router lists 1,281 tokens a second at 182 ms first token on GPT OSS 120B; Developer tier has no hourly or daily caps.\n- **GroqCloud (Groq LLC)**: Hugging Face router lists gpt-oss-20b on Groq at 713 tokens a second with 288 ms first token and gpt-oss-120b at 426; acc-12 placed Groq second to Cerebras. Developer plan limits cover 100 calls; the free plan's 30 a minute does not.\n- **Hugging Face Inference Providers (Hugging Face, Inc.)**: The router publishes throughput and first-token latency per provider and :fastest routes to the highest throughput; the list shows GPT OSS 120B on Cerebras at 1,281 tokens a second and 182 ms first token.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Cloudflare Workers AI (Cloudflare, Inc.)**: A -fast variant of Llama 3.3 70B exists, but no time-to-first-token or tokens-per-second figure is published and text generation is capped at 300 requests a minute.\n- **Fireworks AI Inference (Fireworks.ai, Inc.)**: Fast and Ultra-fast (Preview) variants exist with their own lower limits, but no time-to-first-token or tokens-per-second figure is published and Fireworks is absent from the Hugging Face router's throughput data.\n- **Novita AI (Novita AI)**: Serves gpt-oss and other small models, but acc-12 measured Novita's throughput at about a seventh of Cerebras's on OpenRouter data, and no speed figures are published.\n- **OpenRouter (OpenRouter, Inc.)**: sort throughput or latency and provider pinning pick fast hosts, but the same model id varies by host (acc-04 scored one host 20 points lower) and acc-12's comparison shows a 7x throughput spread.\n- **Prism Inference (Prism Technologies Inc)**: Reasoning can be turned off to cut latency, but no time-to-first-token or throughput figure is published and per-key rate limits are unpublished.\n- **SambaCloud (SambaNova Systems, Inc.)**: Served on SambaNova's RDUs, but no time-to-first-token or tokens-a-second figures in the docs read, and 60 RPM on gpt-oss-120b (Developer tier) is below 100 quick calls.\n- **Together AI Inference (Together Computer, Inc.)**: Serverless targets TPS and TTFT goals but is best effort with 429 or 503 under demand; acc-12 ranks Together's gpt-oss throughput behind Cerebras; no figures published.\n- **Vercel AI Gateway (Vercel Inc.)**: -fast model slugs route to a faster serving path at higher cost with fallback to the base model; no time-to-first-token or token-a-second figures and no published rate limits.\n\n## Couldn't check\n\nThe docs were closed to us, or silent. These carry no mark either way.\n\n- **OpenAI API (OpenAI OpCo, LLC)**: Fast and Ultrafast modes exist per the pricing notes, but no latency or throughput figures were in the pages read.\n- **Simplismart (Verute Technologies Private Limited)**: No latency figures or rate limits in the docs read.\n- **StepFun API (STEPFUN TECHNOLOGY (HK) LIMITED)**: The host platform.stepfun.ai closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read.\n- **Tencent Cloud TokenHub (Tencent Cloud Computing (Beijing) Co., Ltd. (腾讯云计算（北京）有限责任公司))**: The host cloud.tencent.com closed the connection on every attempt (curl exit 56, no HTTP status) from this session, so no page was read.\n\n- not this job: Amazon Bedrock (Amazon Web Services, Inc.): Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published.\n- not this job: Claude API (Anthropic, PBC): Not an open-weight fast host; fast mode on Opus 5.5 is a 2.5x throughput premium option with no published time to first token or tokens-per-second figure.\n- not this job: Antseed (Antseed Foundation): Speed depends on the chosen seller; no latency or throughput figures are published.\n- not this job: Azure OpenAI in Foundry Models (Microsoft Corporation): Not a fast open-weight host; no time-to-first-token or tokens-per-second figures published.\n- not this job: BlockRun.AI (BlockRun, Inc.): Not a fast host; a Nemotron Bolt model is described as about a second with no time-to-first-token or tokens-per-second figure.\n- not this job: Cloudflare AI Gateway (Cloudflare, Inc.): A proxy adds a hop and publishes no latency figures.\n- not this job: Cohere Chat API (Command models) (Cohere Inc.): Not a fast host; no latency or throughput figures published.\n- not this job: DeepInfra (Deep Infra Inc.): The same gpt-oss-120b weights run at 46 tokens a second here against 1,281 on Cerebras in the Hugging Face router's listing, far below the 500 the use case needs.\n- not this job: DeepSeek API (Hangzhou DeepSeek Artificial Intelligence Co., Ltd.): Not a fast host; 36 status incidents and 503 overloads were recorded in the listing, and no latency figures are published.\n- not this job: Eden AI (Eden AI SAS): A gateway hop with no latency figures and a 10 requests a second cap.\n- not this job: Glimpse (Elorian AI, Inc.): A visual reasoning model whose reasoning tokens bill as output; no latency or throughput figures.\n- not this job: Gemini Developer API (Google LLC): Not an open-weight fast host; no time-to-first-token or tokens-per-second figures.\n- not this job: Jais 3 (Mohamed bin Zayed University of Artificial Intelligence (Institute of Foundation Models), with Inception42 (a G42 company)): No published latency or throughput; speed is the owner's hardware.\n- not this job: LiteLLM (Berrie AI Incorporated): A proxy hop with no latency figures.\n- not this job: Meta Model API (Meta Platforms, Inc.): Reasoning models that always reason; no latency or throughput figures.\n- not this job: MiniMax API (Nanonoble Pte. Ltd.): No latency or throughput figures, and M3.1 Flash cannot turn thinking off (400 on disabled or effort none), so every short call pays reasoning tokens.\n- not this job: Mistral AI API (Mistral AI (RCS Paris 952 418 325)): No latency or throughput figures; rate limits are per-organisation numbers shown only in the console.\n- not this job: Muse Spark Open (Meta Platforms, Inc.): 816B mixture of experts with mandatory reasoning at xhigh by default; no time-to-first-token or throughput figures.\n- not this job: Nemotron 3.5 Super VL (NVIDIA Corporation): 12B active parameters but no time-to-first-token or throughput figures, reasoning is on by default, and the trial rate varies with other users' load.\n- not this job: Router.com (Ramp Router) (Ramp Business Corporation): Flex-tier routing is on by default and trades latency for cost; no time-to-first-token or throughput figures.\n- not this job: Beam 1.0 (Reflection AI): Reasoning is mandatory and defaults to max; no time-to-first-token or throughput figures.\n- not this job: RouterPlus (EVO HQ Inc): No time-to-first-token or throughput figures; the Optimize model search is paused.\n- not this job: SiliconFlow (SiliconFlow Labs Pte. Ltd.): No speed figures; a hosted open-weight catalogue, not a fast host.\n- not this job: SWE-2 Flash (Cognition AI, Inc.): No API, so no way to call it per message.\n- not this job: Vertex AI (Gemini Enterprise Agent Platform) (Google LLC): A cloud platform; no time-to-first-token or throughput commitment outside Provisioned Throughput.\n- not this job: xAI API (Grok) (SpaceXSI (xAI)): No time-to-first-token or tokens-a-second figures; priority processing is priced in the pricing notes without speed numbers, and RPS is RPM divided by 48.\n\n6 listings in the category were added after this reading and aren't marked yet.\n\n## What stops agents finishing\n\n- tokens-per-minute caps on the free and first paid tier below what an agent loop sends\n- fast hosts serve a short model list and retire models with little notice\n- same model id served at different speed and quality by different hosts\n\n## Around it\n\nBefore: speech.stt. After: speech.tts, inference.tool-loop.\n\n## Where this comes from\n\nWeight 4 of 100 in Model APIs \u0026 inference: 1 first-hand comparison of 30 prompts x 5 runs plus provider benchmark tables; no Reddit or HN thread reached\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/inference.fast-small-calls.json, or the MCP tool get_use_case with id inference.fast-small-calls. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Many small latency-sensitive calls on fast hardware",
        "url": ""
      }
    ],
    "description": "Many small latency-sensitive calls on fast hardware: 3 companies fully do it, 8 with a limit, 58 carry the tag. Every listing in Model APIs \u0026 inference with the reason and a source.",
    "facts": null,
    "h1": "Many small latency-sensitive calls on fast hardware",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/inference.fast-small-calls/",
    "published": "",
    "section": "agents",
    "title": "Many small latency-sensitive calls on fast hardware | Anchor Terminal",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/inference.fast-small-calls/"
  },
  "tokens": {
    "markdown": 2450,
    "slim": 380
  },
  "version": 1
}
