{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/",
    "json": "https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/index.min.md"
  },
  "markdown": "Two or three companies' docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 3 | 17 | 24 | 14 | 7 |\n\n## What counts as done\n\nAn OpenAI-compatible /v1/chat/completions URL answers correctly after the launch call, with model weights loaded from a template or one install step, and the machine is destroyed when done.\n\nMeasured, for our own tests: the URL answers within 15 minutes of the launch call, and the machine is destroyed by one call.\n\nHow agents ask for it: spin up gpt-oss-20b on an H100 with vLLM and give me the URL; run Llama 4 Scout on 4 GPUs behind an API; host this model privately for my agents instead of paying per token\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **Baseten (Baseten Labs, Inc.)**: Model Library presets serve named Hub models through vLLM's OpenAI-compatible server on a stated GPU, and BIS-LLM deployments expose /v1/chat/completions; deployments are deleted through the Management API. BIS-LLM itself needs support to enable.\n- **Hugging Face Inference Endpoints (Hugging Face, Inc.)**: Pick a Hub model, a cloud instance and the vLLM, SGLang or TGI engine and the endpoint serves an OpenAI-compatible API; tensor_parallel_size defaults to the instance's GPU count, instances run to 8x H200, and the management API deletes the endpoint.\n- **Koyeb (Koyeb SAS)**: The vLLM One-Click app deploys an OpenAI-compatible server on a GPU instance with the Hub model set in the command args, HF_TOKEN and VLLM_API_KEY as environment variables; GPU instances run from 20 GB to 8x H200, and a service is deleted by one API call.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Beam (Smartshare, Inc.)**: The SGLang example serves Qwen2.5-7B-Instruct on an H100 Pod with an OpenAI-compatible API, but it is a code example, not a one-step template: the agent writes app.py with the image and server command. Pods stop after 600 s idle by default.\n- **Cerebrium (Cerebrium Inc.)**: Every endpoint can answer /chat/completions and a vLLM example exists, but the agent writes the handler and image itself; there is no one-step template taking only a model name. A100 and above need the Standard plan.\n- **Cloud Run GPUs (Google LLC)**: An Ollama or vLLM container on an L4 serves an OpenAI-compatible API and a user found spin-up fast and the service easy (acc-35), but the agent builds the image itself and only 24 GB or 96 GB single-GPU instances exist, so models above about 70B do not fit.\n- **CoreWeave (CoreWeave, Inc.)**: Dedicated Inference deployments run dynamo-vllm or vllm behind an OpenAI-compatible endpoint, but weights must first be uploaded to an AI Object Storage bucket (no Hub pulls), the API is v1alpha1, and access is sales-gated.\n- **Crusoe Cloud (Crusoe Technologies LLC)**: Self-serve deployments expose an OpenAI-compatible gateway for a listed base model or a Hugging Face LoRA adapter, billed per GPU-hour; an arbitrary Hub model goes on a VM where the agent installs vLLM over SSH.\n- **GPUsMarket (Tyga.Cloud Ltd)**: Renting a listing or deploying a pod with models [\"llama3.3\"] returns a per-rental token and an OpenAI-compatible /v1/chat/completions proxy; stop ends billing. Prompts pass through the host's machine and the reference says not to send confidential data.\n- **Hyperbolic (Hyperbolic Labs, Inc.)**: Ubuntu with drivers and Docker preinstalled; a user rented 4x H100, pip-installed vLLM over SSH and served Llama 4 Scout with TP=4 (acc-44). No vLLM or Ollama template, so the engine and weights are installed by hand, and the VM bills hourly until terminated.\n- **Lambda Cloud (Lambda, Inc.)**: Lambda Stack ships CUDA, PyTorch and Docker and guides cover vLLM and Ollama, but the agent installs the engine and weights over SSH; there is no serving template and the VM bills per minute until terminated.\n- **Massed Compute (Massed Compute, Inc.)**: A VM with a chosen image and SSH can run vLLM or Ollama, but no engine template or OpenAI-compatible preset appears in the API or MCP tool list; the agent installs the server itself and the VM bills until terminated.\n- **Modal (Modal Labs, Inc.)**: The vLLM example serves a Hub model behind /v1/chat/completions on H200s with scaledown_window set to 15 minutes, but it is a Python app the agent copies and deploys, not a one-step template; cold starts are tens of seconds to minutes, about ten from cache.\n- **Nebius AI Cloud (Nebius B.V.)**: A Serverless AI endpoint runs a vLLM container on a gpu-h100-sxm preset behind a managed HTTPS URL with a token, and stop or delete ends charges; the agent supplies the container and model settings, and there is no engine template taking only a model name.\n- **Northflank (Northflank Ltd)**: The host-a-model guide deploys any serving image (vLLM, Ollama) as a GPU service with ports, secrets and health checks, but the agent supplies the image and command; no engine template takes a model name.\n- **Runpod (Runpod, Inc.)**: The vLLM worker serves a Hub model with OpenAI-compatible responses from a template, but its hub defaults (MAX_NUM_SEQS 256) made an 8B model OOM on an RTX 5090, relaunch in eager mode at half throughput and restart on the same footprint (acc-09).\n- **Thunder Compute (Thunder GPU, Inc.)**: tnr create --gpu a100xl --template ollama gives an instance with Ollama preinstalled and guides run DeepSeek R1 70B and GPT-OSS 120B in minutes, but the model is pulled by hand over SSH and no vLLM template or OpenAI-compatible URL is documented.\n- **Together GPU Clusters (Together Computer, Inc.)**: A Kubernetes cluster can run vLLM, but the agent writes the manifest, the smallest unit is 8 GPUs, and provisioning takes 10 minutes to an hour; no engine template or OpenAI endpoint is provided by the product.\n- **Vast.ai (Vast.ai Inc.)**: A recommended vLLM template plus a search filter gets a model server on a rented H100, which a user did to measure gpt-oss-20b refusal rates (acc-21). Prompts pass through the host's machine unless Secure Cloud is filtered, and the instance bills until destroyed.\n- **Verda (DataCrunch Oy)**: The Serverless Containers vLLM quickstart serves a Hub model behind an OpenAI-compatible endpoint from a container image on an L40S; on a VM the images are Ubuntu or JupyterLab and the engine is installed by hand. Deleting the deployment or instance stops charges.\n\n- not this job: Replicate Deployments (Replicate, LLC): A deployment serves a Cog model behind Replicate's prediction API; no OpenAI-compatible /v1/chat/completions route or vLLM template appears in the docs index, so the success criterion cannot be met.\n\n## What stops agents finishing\n\n- no template for the engine, so the agent must install CUDA, vLLM and the weights over SSH\n- peer host hardware with firmware-capped clocks or that cannot pull images\n- prompts and outputs pass through an individual host's machine, so private data is out\n- an idle machine bills by the hour until destroyed, with no scale to zero\n\n## Around it\n\nBefore: gpu-compute.find-and-provision, open-weights.big-model-on-big-box, open-weights.tool-calling-assistant. After: inference.tool-loop, inference.harness-compat, agent-harnesses.local-model, decision-models.run-on-own-hardware.\n\n## Where this comes from\n\nWeight 14 of 100 in GPU \u0026 serverless compute: 7 of 48 accounts stand up vLLM or Ollama on a rented machine to serve a model to their own tools; two of them left for local hardware on cost.\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/gpu-compute.self-hosted-llm-server.json, or the MCP tool get_use_case with id gpu-compute.self-hosted-llm-server. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Run an open model as my own OpenAI-compatible server for a while",
        "url": ""
      }
    ],
    "description": "Run an open model as my own OpenAI-compatible server for a while: 3 companies fully do it, 17 with a limit, 24 carry the tag. Every listing in GPU \u0026 serverless compute with the reason and a source.",
    "facts": null,
    "h1": "Run an open model as my own OpenAI-compatible server for a while",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/gpu-compute.self-hosted-llm-server/",
    "published": "",
    "section": "agents",
    "title": "Run an open model as my own OpenAI-compatible server for a while |…",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/"
  },
  "tokens": {
    "markdown": 2150,
    "slim": 380
  },
  "version": 1
}
