# LiteLLM (slim)

> Open-source Python SDK and self-hosted AI gateway from BerriAI that puts 100+ model providers behind one OpenAI-compatible API, with virtual keys, spend tracking, budgets, fallbacks, guardrails and an MCP gateway.

- Full: https://www.anchorterminal.com/tools/litellm.md (~8,050 tokens) · this version ~1,830 tokens · JSON https://www.anchorterminal.com/tools/litellm.json · canonical https://www.anchorterminal.com/tools/litellm
- Index: https://www.anchorterminal.com/llms.txt · API: https://www.anchorterminal.com/api/v1/index.json · MCP: https://www.anchorterminal.com/mcp · Updated: 2026-11-07 · pricing https://www.anchorterminal.com/tools/litellm#pricing · reviews https://www.anchorterminal.com/tools/litellm#reviews
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)

**B · 62.4/100 · rank #557 of 1486 · #20 in Model APIs & inference · not agent-ready · confidence medium**


Assessment: An MIT-licensed gateway that puts 100+ model providers behind one OpenAI-compatible API, with virtual keys, budgets, fallbacks and an MCP gateway, released several times a week. Its security record is the main limitation. Two PyPI releases carried a credential stealer on 24 March 2026, and 30 GitHub advisories followed in 12 months, five of them critical.

## Facts

- Kind: Model router · vendor: BerriAI · category: Model APIs & inference · legal entity: Berrie AI Incorporated · provenance 49/100
- Local only (HTTP): pypi `litellm`, docker `docker.litellm.ai/berriai/litellm`
- Auth: API key · pricing: Freemium · x402: no · licence: MIT, except the enterprise/ directory, which is under its own licence and needs an Enterprise licence key
- Benchmark probes: not run yet, so Performance is pending
- Surfaces: Python SDK (`litellm.completion()` and async versions) and a self-hosted gateway on port 4000 with OpenAI, Anthropic Messages, MCP and A2A endpoints, an Admin UI at /ui and Postgres for keys and spend
- Routing: Load balancing, retries, cooldowns and fallbacks across deployments and providers, tag-based routing and an auto router
- Credentials: Master key with full admin rights. Virtual keys with model, route, MCP server, budget, rate and expiry limits, blockable. Rotation with a grace period is Enterprise only
- Errors: OpenAI-shaped `error` envelope with a string `code`. Gateway 429s carry `retry-after`, `rate_limit_type` and `reset_at`; provider 429s don't. Management API answers RFC 9457 problem+json
- MCP gateway: One endpoint for registered MCP servers with access by key, team or organisation, OAuth patterns, guardrails before and after tool calls, and a tool-search mode that exposes four virtual tools
- Telemetry: None by default. Usage telemetry (v1.107.0 and later) counts requests and Admin UI page views only when the operator turns groups on and picks their own table or HTTP endpoint, with `LITELLM_TELEMETRY_DISABLED=true` to lock it off. Start-up fetches the model price map from GitHub unless `LITELLM_LOCAL_MODEL_COST_MAP=True`
- Release support: The four most recent stable minor lines get patches. A new minor line ships roughly weekly. Release notes carry a breaking changes section
- Image security: GHCR images signed with cosign; the signing key changed in the 24 October 2026 patch releases. No SBOM or provenance attestation, and the default image runs as root (a `-non_root` variant exists)
- Enterprise: Annual licence through sales. SSO beyond five users, SCIM, audit logs, key rotation, IP allowlists, team-scoped guardrails, 72-hour security patch SLA. 30-day trial with no card
- Certifications: SOC 2 Type II per the data security page, report through trust.litellm.ai

- Scores: Reliability 70, Performance pending, Schema & documentation 91, Agent ergonomics 79, Security & auth 72, Payments & pricing 60, Task success pending, Maintenance & community 82, Transparency & trust 63 · negative events -12 · total over the 7 assessed categories
- Why: Reliability, Read with the local-software lines, since the gateway runs on the operator's hardware and BerriAI hosts nothing an agent calls. · Schema & documentation, Every gateway serves Swagger at /, ReDoc at /redoc and the raw schema at /openapi.json, and the docs host a public Swagger explorer (25). · Agent ergonomics, Read with the API lines for the gateway an agent calls. · Security & auth, Read with the tool checklist for self-hosted software. · Payments & pricing, Read with the self-hosted rule, since an agent uses the free gateway and Enterprise is a sales-led licence for the same software. · Maintenance & community, v1.108.1 tagged on 5 November 2026, v1.108.0 and v1.106.7 on 3 and 4 November (30). · Transparency & trust, MIT for everything outside enterprise/, which has its own licence (25 of 30).
- Sources: 23, open questions: 5, both in the full twin
- Capabilities: inference.router, obs.gateway, agent.mcp-client, guard.pii
- JSON: https://www.anchorterminal.com/api/v1/tools/litellm.json
- Verify (for the vendor): the badge `https://www.anchorterminal.com/badges/litellm.svg` or a link to https://www.anchorterminal.com/tools/litellm from a page on litellm.ai or one of its subdomains, or the README at the root of github.com/BerriAI/litellm, then `POST https://www.anchorterminal.com/api/v1/verify` `{"slug", "url"}` or `verify_listing` at /mcp; re-checked weekly, no effect on the grade. Snippets in the full twin.
- Support: Discord https://discord.com/invite/wuPM9dRgDw; support@berri.ai; https://docs.litellm.ai; https://github.com/BerriAI/litellm/issues; more in the full twin

## Before you call it

1. Pin an exact litellm version or image digest, verify the cosign signature, and never install an unpinned latest in CI
2. Give each agent a virtual key limited to its models, routes and MCP servers, with a budget and an expiry
3. Treat a 429 with `retry-after` as the gateway's own limit and wait that long; a 429 without it came from the provider
4. Set `NO_DOCS`, `NO_REDOC` and `NO_OPENAPI` to True in production, and use the `-non_root` image variant
5. Set `LITELLM_LOCAL_MODEL_COST_MAP=True` where egress is blocked, since start-up otherwise fetches the price map from GitHub

## Connect

```bash
pip install litellm   # SDK; gateway: curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker/docker-compose.quickstart.yml && docker compose -f docker-compose.quickstart.yml up -d
```

```bash
curl http://localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-<your-virtual-key>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-5.6-terra",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'
```

```bash
export ANTHROPIC_BASE_URL="http://localhost:4000"
export ANTHROPIC_AUTH_TOKEN="sk-<your-virtual-key>"
export ANTHROPIC_MODEL="<a model_name from your config>"
claude
```

## Similar tools

| Tool | Grade | Score | Shared capabilities | Slim |
| --- | --- | --- | --- | --- |
| Cloudflare AI Gateway | B | 67.8 | inference.router, obs.gateway, guard.pii | https://www.anchorterminal.com/tools/cloudflare-ai-gateway.min.md |
| Vercel AI Gateway | B | 67.8 | inference.router, obs.gateway | https://www.anchorterminal.com/tools/vercel-ai-gateway.min.md |
| LocalAI | B | 62.5 | agent.mcp-client, guard.pii | https://www.anchorterminal.com/tools/localai.min.md |
| Tencent Cloud TokenHub | C | 59.5 | inference.router, obs.gateway | https://www.anchorterminal.com/tools/tencent-tokenhub.min.md |
| AI SDK | AA | 89.8 | agent.mcp-client | https://www.anchorterminal.com/tools/vercel-ai-sdk.min.md |


## Panel reviews (0, desk reviews from public material, no calls made)