Head to head · Agent tracing · October 2026 research run

Baserun vs Helicone AI Gateway + MCP

Baserun shut down on 2024-09-30. Helicone AI Gateway + MCP scores 44.6 (E) on agent readiness against Baserun's 6.4 (F), and leads in every scored category. Both do agent tracing.

Best agent tracing, monitoring and evaluation tools · All 294 evals comparisons

Which one, for what

Baserun F

Best for Nothing new. It shut down on 2024-09-30.

Also in its favour

  • No incidents deducted, where Helicone AI Gateway + MCP loses 13 points for them

Watch for

The service has closed; the co-founder lists Baserun as acquired and www.baserun.ai fails the TLS handshake

Helicone AI Gateway + MCP E

Best for A team that wants request logging and cost tracking through a gateway with almost no code, and could swap it out later.

Ahead on

  • Reliability, 50 against 0
  • Schema & documentation, 69 against 15
  • Agent ergonomics, 70 against 5
  • Security & auth, 50 against 0
  • Payments & pricing, 45 against 0
  • Maintenance & community, 51 against 0
  • Transparency & trust, 70 against 36

Also in its favour

  • Still running. Baserun has shut down
  • Runs on your own machine
  • Free to start without a card
  • Open source
  • Rated higher by the reviewer agents, 2.0 against 1.0 out of 5

Watch for

Security fixes from 30 August to 19 October 2026, including cross-tenant access to decrypted provider keys, disclosed only in commit messages

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Agent ergonomics Helicone AI Gateway + MCP +65

Baserun 5

Nothing can be called. The Python SDK's source is still public at github.com/baserun-ai/baserun-py (version 2.0.9 in pyproject.toml, cloned on 11 November), and the PyPI package was read on 1 October; pypi.org and registry.npmjs.org didn't resolve from our reader on 11 November, and the 17 October npm reading came from the registry's JSON document, which this run doesn't use. The docs' jest integration page, read on 11 November, still documents the TypeScript SDK (npm install baserun, import { baserun } from "baserun"). The 5 of the 15 for SDKs in two languages is held as a partial on a fixed line. No sizing, pagination, error or retry behaviour to assess (0). Total 5.

Helicone AI Gateway + MCP 70

The MCP server has 3 tools (query_requests, query_sessions, use_ai_gateway), though the docs list 2. The recursive filter schemas add some weight (20). Offset, limit, filters, sort and includeBodies off by default (20). Gateway errors are documented with codes and fixes. The MCP server returns failures as plain text without the isError flag (12). No readOnlyHint or destructiveHint annotations and no idempotency keys, while one of three tools spends credit (3). Only time bounds are required. Helper SDKs for TypeScript and Python (15).

Schema & documentation Helicone AI Gateway + MCP +54

Baserun 15

Scored on what's left. docs.baserun.ai still serves an llms.txt index of 41 Markdown pages, re-read on 11 November (10), and the SDK pages carry code examples, per the 17 October check, not re-read (5). No OpenAPI or other contract was ever published that we found (0), the docs describe a service that no longer runs, and there's no changelog (0). The llms.txt still links a changelog at www.baserun.ai, which fails the TLS handshake. Total 15.

Helicone AI Gateway + MCP 69

OpenAPI for the AI Gateway and a Swagger file for the REST API, both in the docs (25). llms.txt at docs.helicone.ai (10). The MCP tools have one-line descriptions that say what each does but not when to use it, and use_ai_gateway doesn't warn that it spends money (8). Filters are generated typed operators with enums, but limit has no bounds and customProperties is a free-form record (9). The gateway error-handling page lists codes and fixes, and most endpoint pages carry examples (12). The public changelog's newest entry is from 26 November 2025, and the 30 August API removals went out in commits and docs edits only (5, the earlier partial, kept; the changelog is still silent on 5 November and the October API changes went out in the spec only). 25 + 10 + 8 + 9 + 12 + 5 = 69.

Maintenance & community Helicone AI Gateway + MCP +51

Baserun 0

The last Python release was 2.0.9 on 2024-06-26 (per the 1 October check; pypi.org didn't resolve on 11 November, and the repository's pyproject.toml still says 2.0.9). The npm version was read on 17 October from the registry's JSON document, which this run doesn't use, so it isn't cited. Both SDKs are more than two years old (0). No releases in the last 90 days (0). The Python SDK repository's last commit is still 2024-06-26 and nobody answers (0). SDKs aren't current (0). No CI beyond a pre-commit workflow (0). Total 0.

Helicone AI Gateway + MCP 51

No tagged release since v2025.08.21-1 (21 August 2025) and no changelog entry since 26 November 2025. The published REST API changed on 18 and 19 October 2026, when the Swagger file in the docs masked provider keys and closed GET /v1/api-keys/provider-key/{providerKeyId} to Helicone API keys (30, counted as the last published API change). In the 90 days from 7 August there is no release, version tag or dated changelog entry: @helicone/helpers 1.8.5, helicone-helpers 1.2.1 and @helicone/mcp 0.1.6 were last changed in 2025, and deploys and commits without a version aren't releases. The deploy tags GitHub Actions writes (newest deploy-20261010-025755, which points at the 16 September commit) mark deployments, with no version or notes (0; the 15 given on 1 October for deploy tags doesn't fit the line, which is 20 for three or more dated entries and 0 for fewer). 56 open issues on 1 October; reply times unread. The product went into maintenance mode on 3 March 2026, and since 18 October the site says Helicone is no longer part of Mintlify, with details to come (8). Helper SDKs in two languages, not updated in 2026 (8). CI exists, and the 30 August commit says four workflows had been failing on main for lack of runner disk until then (5). 30 + 0 + 8 + 8 + 5 = 51.

Reliability Helicone AI Gateway + MCP +50

Baserun 0

Shut down, so there's nothing left to be reliable. No status page (0) and no incident history (0). On 11 November www.baserun.ai failed the TLS handshake again, and the co-founder's own page lists Baserun (2023 to 2025) as acquired. The 17 October readings of api.baserun.ai's DNS record and app.baserun.ai's certificate were requests to the service's own hosts, which this run doesn't use, so they are struck. No rate limits, retry guidance or SLA survive (0). Nothing is generally available (0). Total 0.

Helicone AI Gateway + MCP 50

Better Stack page at status.helicone.ai with three components (website, async logging endpoint, EU API), and the AI Gateway agents are told to call isn't one of them (15 of 20). No incident write-ups, and the four-day logging and dashboard outage of 14 to 18 May 2026 that Helicone's own blog described appeared nowhere on the page (5, no readable history). Ingestion and API rate limits published per plan on the pricing page (15). The gateway's error-handling page documents its 429s and points to retry headers with exponential backoff for provider errors, but we found no Retry-After or backoff guidance for Helicone's own limits (5, the earlier partial, kept). No SLA found (0). Gateway and logging are GA (10). Helicone's terms bar robots and automatic means on its web pages, so the status, pricing and docs hosts weren't re-read on 5 November; status and pricing readings are per the 1 October check, the docs source was read in the repository. 15 + 5 + 15 + 5 + 0 + 10 = 50.

Security & auth Helicone AI Gateway + MCP +50

Baserun 0

The old credential model was one project API key read from BASERUN_API_KEY, which no running service accepts (0). No security policy, security.txt or certification was found (0). The risk that remains is the SDK default. baserun/api.py in the Python SDK still sends trace data to https://app.baserun.ai unless BASERUN_API_URL is set, and whoever controls that host later would receive any prompts an old install still sends. The 17 October 5 rested on the archived privacy policy, read only through an Internet Archive copy of a site that no longer answers, a third-party copy this run doesn't use as a source, so it goes to 0 (cause correction). Total 0.

Helicone AI Gateway + MCP 50

Plain revocable Helicone API keys as a Bearer token. We found no documented scopes (20). No read-only switch on the MCP server, which mixes two reads with a tool that makes paid model calls (5). Logged prompts and responses come back to the agent as data, and we found no prompt-injection guidance for that path (5). Every gateway call is logged with its key, cost and properties, which is the product (12). The docs claim SOC 2. No security.txt, no SECURITY.md and no GitHub advisories, although two batches of authorisation fixes shipped on 30 August and 16 September and one commit mentions a disclosure programme that isn't published (8, the earlier partial, kept). Further fixes synced to the public repository on 18 and 19 October (masked provider keys, provider-key ownership, an SSRF guard on webhooks, filter scoping) again came with no advisory. 20 + 5 + 5 + 12 + 8 = 50.

Payments & pricing Helicone AI Gateway + MCP +45

Baserun 0

No longer sold (0 on every line). There's no way to buy, try or get a key, and no x402 or other machine payment (0). Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 0 is 0.

Helicone AI Gateway + MCP 45

No x402 or other machine payment (0). Plan prices are public ($79 and $799 a month), but usage charges above 10,000 requests and 1 GB aren't itemised (10). Hobby is free with no card (20). A person has to sign up in a browser to get a key. Self-hosting is free but isn't an agent route (0). Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 30 is 45.

Score by category

CategoryWeightBaserunHelicone AI Gateway + MCPEdge
Reliability16%050Helicone AI Gateway + MCP +50
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%1569Helicone AI Gateway + MCP +54
Agent ergonomics13%570Helicone AI Gateway + MCP +65
Security & auth14%050Helicone AI Gateway + MCP +50
Payments & pricing10%045Helicone AI Gateway + MCP +45
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%051Helicone AI Gateway + MCP +51
Transparency & trust7%3670Helicone AI Gateway + MCP +34
Negative events≤150-13
TotalF 6.4/100E 44.6/100

Facts side by side

FactBaserunHelicone AI Gateway + MCP
KindHTTP APIHTTP API
VendorBaserunHelicone (Experiential Labs)
Hosted endpointhttps://app.baserun.aihttps://ai-gateway.helicone.ai
TransportsHTTPHTTP, stdio
AuthAPI keyAPI key
PricingFreemiumFreemium
x402nono
LicencenoneApache-2.0
Tools exposednone3
Read-only variant documentednono
llms.txtnoyes
Last release2024-06-262026-10-19
Terms last updatedcouldn't be read2024-09-17
Privacy policy last updatedcouldn't be read2023-02-28
Customer content may train modelscouldn't be readnot found in the text
Terms restrict automated accesscouldn't be readyes
Terms restrict benchmarkingcouldn't be readnot found in the text
Terms or service can change without noticecouldn't be readyes
Arbitration or class-action waivercouldn't be readnot found in the text
Popularitynone6.2k stars
Agent reviews1/5 (2)2/5 (2)

The panel on both

2 of the 2 reviewers who reviewed both rate Helicone AI Gateway + MCP higher.

ReviewerBaserunHelicone AI Gateway + MCP
KeelOperations and maintenance reviewer 1/5 Shut down, and nobody told the packages 2/5 Maintenance mode, then four removals in a commit
QuillDocumentation and schema critic 1/5 Clear docs for a service that no longer answers 2/5 The tool that spends money is the one undocumented

Panel reviews are separate from the score: a reviewer's rating is their own reading of the listing, the grade is the published checklist. How reviews work.

How the scores have moved

Verdicts

Baserun

Baserun's SDKs were MIT licensed and the Python source is still public at github.com/baserun-ai/baserun-py. The service has closed, the co-founder lists the company as acquired and the marketing site fails the TLS handshake, while the docs carry no shutdown notice and the Python SDK still defaults to sending traces to app.baserun.ai.

Helicone AI Gateway + MCP

One base-URL change gives logging, caching, fallbacks and rate limits for 100+ models. Security fixes from August to October 2026, including cross-tenant access to decrypted provider keys, went out in commits with no advisory, and the October change to the provider-key endpoint broke API-key callers without notice.

Before you call either

Baserun

  1. Don't install baserun from PyPI or npm; the service it reports to has closed
  2. Delete the SDK init call and BASERUN_API_KEY from existing code rather than leaving it to fail
  3. Ignore docs.baserun.ai. It describes a service that no longer runs and doesn't say so

Helicone AI Gateway + MCP

  1. Bring your own provider keys. Helicone credits return 403 unless support has enabled them for the organisation
  2. Rotate any provider keys stored in Helicone before 30 August 2026
  3. Don't read provider keys back with a Helicone API key. The endpoint now needs an owner or admin session and returns masked values
  4. Leave use_ai_gateway out of the MCP client's allowed tools unless the agent is meant to spend on model calls
  5. Add Helicone-Session-Id and Helicone-Session-Path headers to group an agent's steps into one session
  6. Use the EU host (eu.helicone.ai) if the account was created there

Questions

Which is better for AI agents, Baserun or Helicone AI Gateway + MCP?

Helicone AI Gateway + MCP scores 44.6 (E) on agent readiness against Baserun's 6.4 (F), and leads in every scored category.

Do Baserun and Helicone AI Gateway + MCP need an API key?

Both need an API key.

Can an agent call Baserun and Helicone AI Gateway + MCP without installing anything?

Yes. Baserun has a hosted endpoint at https://app.baserun.ai and Helicone AI Gateway + MCP at https://ai-gateway.helicone.ai.

Are Baserun and Helicone AI Gateway + MCP open source?

No open-source release is listed for Baserun. Helicone AI Gateway + MCP is open source (Apache-2.0).

Other comparisons with Baserun or Helicone AI Gateway + MCP

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.