{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/gpu-compute.serverless-model-endpoint/",
    "json": "https://www.anchorterminal.com/niches/gpu-compute.serverless-model-endpoint/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/gpu-compute.serverless-model-endpoint/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/gpu-compute.serverless-model-endpoint/index.min.md"
  },
  "markdown": "No company's docs show a full fit yet. The listings below do the job with a limit, or not at all.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 0 | 11 | 24 | 28 | 21 |\n\n## What counts as done\n\nA first request after scale to zero returns a correct answer including weight load, a warm request returns, and the endpoint bills nothing after the idle window with no replica left running.\n\nMeasured, for our own tests: the first request after scale to zero answers within 60 seconds, and a warm request within 2 seconds plus inference.\n\nHow agents ask for it: deploy this model as an API I can call; run Whisper or a diffusion model behind an endpoint that costs nothing when idle; host my fine-tune so my app can call it\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Baseten (Baseten Labs, Inc.)**: min_replica 0 scales to zero and a sync request is parked until a replica is ready (1200 s default, then 500). Replica time bills per minute from start-up, and scale_down_delay defaults to 900 s, so each burst bills 15 idle minutes. No cold-start figure is published.\n- **Beam (Smartshare, Inc.)**: Endpoints scale to zero and bill only while a container runs; cold start and image pull are free. The default keep_warm_seconds of 180 bills three idle minutes after every call unless lowered, the endpoint budget is capped at 600 s, and no cold-start figure is published.\n- **Cerebrium (Cerebrium Inc.)**: min_replicas 0 scales to zero, cooldown defaults to 60 s, and cold-start time is not billed while model initialisation is. A request queues until a container is ready. disable_auth defaults to true, so the endpoint is public until changed, and no cold-start figure is published.\n- **Cloud Run GPUs (Google LLC)**: GPU services scale to zero and the GPU is usable about 5 s after an instance starts, with weights loaded on top from Cloud Storage. Instance-based billing charges the whole instance lifetime, idle included, so a request every 15 minutes billed a user all day (acc-36). One L4 or RTX PRO 6000 per instance.\n- **Hugging Face Inference Endpoints (Hugging Face, Inc.)**: Scale to zero defaults to 1 hour idle (settable to 15 minutes) and bills nothing at zero replicas, but the docs say scaling from zero takes minutes, the proxy answers 503 unless X-Scale-Up-Timeout holds the request, and users reported endpoints stuck Initializing for a week (acc-39) and no 503 on wake (acc-40).\n- **Koyeb (Koyeb SAS)**: GPU services scale to zero (public preview) after a 5 minute idle period and wake on the next request with a 1 to 5 s VM cold start, plus model load. Light Sleep snapshots are CPU only, the idle period is only adjustable on Pro and above, and no model-level cold-start figure is published.\n- **Modal (Modal Labs, Inc.)**: Functions scale to zero by default with a 60 s scaledown_window and the web request waits through the cold start. Users measured 20 to 30 s for demucs (acc-13) and about a minute for ComfyUI (acc-12); GPU memory snapshots are alpha and one user saw 24 of 41 restores segfault (acc-18). Endpoints are public until a proxy token is added.\n- **Replicate Deployments (Replicate, LLC)**: POST /v1/deployments with hardware, min_instances 0 and max_instances gives a private autoscaling endpoint that scales to zero, but instances bill per second for set-up and idle time, prediction data is deleted after an hour, and no cold-start figure or scale-down window is published.\n- **Runpod (Runpod, Inc.)**: Workers scale to zero after an idle timeout (5 s console default) and FlashBoot caches starts, but workers bill during cold start and idle. Users measured 63 s engine-ready for Qwen3-8B (acc-11), 12.7 s to first byte for a TTS template (acc-05) and cold starts up to five minutes (acc-06).\n- **Vast.ai (Vast.ai Inc.)**: Serverless endpoints recruit marketplace workers from a template with min_workers, inactivity_timeout and cold_mult parameters and pay only for execution, but each new endpoint needs a benchmarking evaluation period across GPU classes, workers are peer hosts, and no cold-start figure is published.\n- **Verda (DataCrunch Oy)**: Serverless Containers scale to zero when idle with queue-based autoscaling and a vLLM quickstart, billing every minute a replica is up including spin-up and spin-down. The scale-down delay must be tuned by hand and no cold-start figure is published.\n\n- not this job: CoreWeave (CoreWeave, Inc.): Dedicated Inference autoscaling requires min at least 1 and states scale-to-zero is not supported; CKS node pools are whole 8-GPU nodes billed by the hour. Nothing bills zero when idle.\n- not this job: Crusoe Cloud (Crusoe Technologies LLC): VMs bill per second while running with no autoscaling, and self-serve deployments reserve GPUs by the hour for Crusoe's catalogue models and LoRA adapters, not a user's container or handler.\n- not this job: GPUsMarket (Tyga.Cloud Ltd): Rentals serve the host's Ollama or OpenAI-compatible models and bill per second until stopped; there is no custom handler or container with scale to zero. Cloud pods can be paused but not autoscaled.\n- not this job: Hyperbolic (Hyperbolic Labs, Inc.): Hourly VM and bare-metal rentals only; no containers, endpoints, autoscaling or scale to zero. The separate Inference product serves Hyperbolic's hosted models, not the user's.\n- not this job: Lambda Cloud (Lambda, Inc.): On-demand VMs billed by the minute until terminated; no containers, endpoints, autoscaling or scale to zero.\n- not this job: Massed Compute (Massed Compute, Inc.): Hourly GPU VMs only; no containers, endpoints, autoscaling or scale to zero, and a stopped VM still bills the full hourly rate.\n- not this job: Nebius AI Cloud (Nebius B.V.): A Serverless AI endpoint is one container on one VM that runs until stopped; stopping bills nothing but there is no autoscaling or scale to zero, so an idle endpoint keeps paying GPU rates until a call stops it.\n- not this job: Northflank (Northflank Ltd): GPU services autoscale between a minimum and maximum instance count but have no scale to zero; a service bills until paused or deleted, so an idle endpoint keeps paying.\n- not this job: Thunder Compute (Thunder GPU, Inc.): Persistent instances billed per minute while they run and sandboxes with a fixed lifetime; no autoscaling endpoint or scale to zero.\n- not this job: Together GPU Clusters (Together Computer, Inc.): Clusters bill hourly from creation until deleted and cannot be paused or scaled to zero; there is no endpoint product here.\n\n## What stops agents finishing\n\n- cold start of a minute or more for a request-response app, so the user gives up before the first token\n- endpoint stuck initialising with no logs and no 503, so the caller waits on nothing\n- a warm replica or keep-warm window bills while idle, so a request every 15 minutes costs a full day\n- snapshot restore that fails intermittently and falls back to a full cold start\n- endpoint public by default until a token is added\n\n## Around it\n\nBefore: gpu-compute.package-and-deploy, open-weights.fine-tune-base. After: inference.tool-loop, speech-to-text.long-recording-batch, voice-agents.low-latency-pipeline-build.\n\n## Where this comes from\n\nWeight 28 of 100 in GPU \u0026 serverless compute: 21 of 48 accounts are about an endpoint on serverless GPU: cold start, idle billing, wake from zero. The most written-about job on Hacker News and in the issue trackers read.\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/gpu-compute.serverless-model-endpoint.json, or the MCP tool get_use_case with id gpu-compute.serverless-model-endpoint. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Serve my own model as an endpoint that scales to zero",
        "url": ""
      }
    ],
    "description": "Serve my own model as an endpoint that scales to zero: 0 companies fully do it, 11 with a limit, 24 carry the tag. Every listing in GPU \u0026 serverless compute with the reason and a source.",
    "facts": null,
    "h1": "Serve my own model as an endpoint that scales to zero",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/gpu-compute.serverless-model-endpoint/",
    "published": "",
    "section": "agents",
    "title": "Serve my own model as an endpoint that scales to zero | Anchor…",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/gpu-compute.serverless-model-endpoint/"
  },
  "tokens": {
    "markdown": 2150,
    "slim": 330
  },
  "version": 1
}
