# Run an open model as my own OpenAI-compatible server for a while

> Run an open model as my own OpenAI-compatible server for a while: 3 companies fully do it, 17 with a limit, 24 carry the tag. Every listing in GPU & serverless compute with the reason and a source.

- Canonical: https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/
- Markdown: https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/index.md (~2,150 tokens)
- Slim: https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/index.min.md (~380 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/niches/gpu-compute.self-hosted-llm-server/index.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-15

Two or three companies' docs show a full fit.

| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |
| --- | --- | --- | --- | --- |
| 3 | 17 | 24 | 14 | 7 |

## What counts as done

An OpenAI-compatible /v1/chat/completions URL answers correctly after the launch call, with model weights loaded from a template or one install step, and the machine is destroyed when done.

Measured, for our own tests: the URL answers within 15 minutes of the launch call, and the machine is destroyed by one call.

How agents ask for it: spin up gpt-oss-20b on an H100 with vLLM and give me the URL; run Llama 4 Scout on 4 GPUs behind an API; host this model privately for my agents instead of paying per token

## Does the job

The docs show it meets the test above, and nothing we read says otherwise.

- **Baseten (Baseten Labs, Inc.)**: Model Library presets serve named Hub models through vLLM's OpenAI-compatible server on a stated GPU, and BIS-LLM deployments expose /v1/chat/completions; deployments are deleted through the Management API. BIS-LLM itself needs support to enable.
- **Hugging Face Inference Endpoints (Hugging Face, Inc.)**: Pick a Hub model, a cloud instance and the vLLM, SGLang or TGI engine and the endpoint serves an OpenAI-compatible API; tensor_parallel_size defaults to the instance's GPU count, instances run to 8x H200, and the management API deletes the endpoint.
- **Koyeb (Koyeb SAS)**: The vLLM One-Click app deploys an OpenAI-compatible server on a GPU instance with the Hub model set in the command args, HF_TOKEN and VLLM_API_KEY as environment variables; GPU instances run from 20 GB to 8x H200, and a service is deleted by one API call.

## Does the job, with a limit

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

- **Beam (Smartshare, Inc.)**: The SGLang example serves Qwen2.5-7B-Instruct on an H100 Pod with an OpenAI-compatible API, but it is a code example, not a one-step template: the agent writes app.py with the image and server command. Pods stop after 600 s idle by default.
- **Cerebrium (Cerebrium Inc.)**: Every endpoint can answer /chat/completions and a vLLM example exists, but the agent writes the handler and image itself; there is no one-step template taking only a model name. A100 and above need the Standard plan.
- **Cloud Run GPUs (Google LLC)**: An Ollama or vLLM container on an L4 serves an OpenAI-compatible API and a user found spin-up fast and the service easy (acc-35), but the agent builds the image itself and only 24 GB or 96 GB single-GPU instances exist, so models above about 70B do not fit.
- **CoreWeave (CoreWeave, Inc.)**: Dedicated Inference deployments run dynamo-vllm or vllm behind an OpenAI-compatible endpoint, but weights must first be uploaded to an AI Object Storage bucket (no Hub pulls), the API is v1alpha1, and access is sales-gated.
- **Crusoe Cloud (Crusoe Technologies LLC)**: Self-serve deployments expose an OpenAI-compatible gateway for a listed base model or a Hugging Face LoRA adapter, billed per GPU-hour; an arbitrary Hub model goes on a VM where the agent installs vLLM over SSH.
- **GPUsMarket (Tyga.Cloud Ltd)**: Renting a listing or deploying a pod with models ["llama3.3"] returns a per-rental token and an OpenAI-compatible /v1/chat/completions proxy; stop ends billing. Prompts pass through the host's machine and the reference says not to send confidential data.
- **Hyperbolic (Hyperbolic Labs, Inc.)**: Ubuntu with drivers and Docker preinstalled; a user rented 4x H100, pip-installed vLLM over SSH and served Llama 4 Scout with TP=4 (acc-44). No vLLM or Ollama template, so the engine and weights are installed by hand, and the VM bills hourly until terminated.
- **Lambda Cloud (Lambda, Inc.)**: Lambda Stack ships CUDA, PyTorch and Docker and guides cover vLLM and Ollama, but the agent installs the engine and weights over SSH; there is no serving template and the VM bills per minute until terminated.
- **Massed Compute (Massed Compute, Inc.)**: A VM with a chosen image and SSH can run vLLM or Ollama, but no engine template or OpenAI-compatible preset appears in the API or MCP tool list; the agent installs the server itself and the VM bills until terminated.
- **Modal (Modal Labs, Inc.)**: The vLLM example serves a Hub model behind /v1/chat/completions on H200s with scaledown_window set to 15 minutes, but it is a Python app the agent copies and deploys, not a one-step template; cold starts are tens of seconds to minutes, about ten from cache.
- **Nebius AI Cloud (Nebius B.V.)**: A Serverless AI endpoint runs a vLLM container on a gpu-h100-sxm preset behind a managed HTTPS URL with a token, and stop or delete ends charges; the agent supplies the container and model settings, and there is no engine template taking only a model name.
- **Northflank (Northflank Ltd)**: The host-a-model guide deploys any serving image (vLLM, Ollama) as a GPU service with ports, secrets and health checks, but the agent supplies the image and command; no engine template takes a model name.
- **Runpod (Runpod, Inc.)**: The vLLM worker serves a Hub model with OpenAI-compatible responses from a template, but its hub defaults (MAX_NUM_SEQS 256) made an 8B model OOM on an RTX 5090, relaunch in eager mode at half throughput and restart on the same footprint (acc-09).
- **Thunder Compute (Thunder GPU, Inc.)**: tnr create --gpu a100xl --template ollama gives an instance with Ollama preinstalled and guides run DeepSeek R1 70B and GPT-OSS 120B in minutes, but the model is pulled by hand over SSH and no vLLM template or OpenAI-compatible URL is documented.
- **Together GPU Clusters (Together Computer, Inc.)**: A Kubernetes cluster can run vLLM, but the agent writes the manifest, the smallest unit is 8 GPUs, and provisioning takes 10 minutes to an hour; no engine template or OpenAI endpoint is provided by the product.
- **Vast.ai (Vast.ai Inc.)**: A recommended vLLM template plus a search filter gets a model server on a rented H100, which a user did to measure gpt-oss-20b refusal rates (acc-21). Prompts pass through the host's machine unless Secure Cloud is filtered, and the instance bills until destroyed.
- **Verda (DataCrunch Oy)**: The Serverless Containers vLLM quickstart serves a Hub model behind an OpenAI-compatible endpoint from a container image on an L40S; on a VM the images are Ubuntu or JupyterLab and the engine is installed by hand. Deleting the deployment or instance stops charges.

- not this job: Replicate Deployments (Replicate, LLC): A deployment serves a Cog model behind Replicate's prediction API; no OpenAI-compatible /v1/chat/completions route or vLLM template appears in the docs index, so the success criterion cannot be met.

## What stops agents finishing

- no template for the engine, so the agent must install CUDA, vLLM and the weights over SSH
- peer host hardware with firmware-capped clocks or that cannot pull images
- prompts and outputs pass through an individual host's machine, so private data is out
- an idle machine bills by the hour until destroyed, with no scale to zero

## Around it

Before: gpu-compute.find-and-provision, open-weights.big-model-on-big-box, open-weights.tool-calling-assistant. After: inference.tool-loop, inference.harness-compat, agent-harnesses.local-model, decision-models.run-on-own-hardware.

## Where this comes from

Weight 14 of 100 in GPU & serverless compute: 7 of 48 accounts stand up vLLM or Ollama on a rented machine to serve a model to their own tools; two of them left for local hardware on cost.

As data: https://www.anchorterminal.com/api/v1/use-cases/gpu-compute.self-hosted-llm-server.json, or the MCP tool get_use_case with id gpu-compute.self-hosted-llm-server. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
