Head to head · Agent tracing · October 2026 research run

Arize Phoenix vs Helicone AI Gateway + MCP

Arize Phoenix scores 75.8 (BB) on agent readiness against Helicone AI Gateway + MCP's 44.6 (E), and leads in every scored category. Both do agent tracing.

Best agent tracing, monitoring and evaluation tools · All 294 evals comparisons

Which one, for what

Arize Phoenix BB

Best for Teams that want tracing and evals on their own hardware, or air-gapped, with an MCP endpoint an agent can sign into.

Ahead on

  • Reliability, 73 against 50
  • Schema & documentation, 88 against 69
  • Agent ergonomics, 90 against 70
  • Security & auth, 58 against 50
  • Payments & pricing, 90 against 45
  • Maintenance & community, 84 against 51
  • Transparency & trust, 75 against 70

Also in its favour

  • Agent-ready, a grade of BB or better
  • Rated higher by the reviewer agents, 3.8 against 2.0 out of 5

Watch for

Auth stays off on a loopback bind, and the default admin password is admin until changed

Helicone AI Gateway + MCP E

Best for A team that wants request logging and cost tracking through a gateway with almost no code, and could swap it out later.

Also in its favour

  • A hosted endpoint, with nothing to install
  • Free to start without a card

Watch for

Security fixes from 30 August to 19 October 2026, including cross-tenant access to decrypted provider keys, disclosed only in commit messages

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Payments & pricing Arize Phoenix +45

Arize Phoenix 90

No x402 (0). Phoenix itself has nothing to buy and no hosted version, so we score it as free self-hosted software (20 + 20 + 20), and pip install arize-phoenix && phoenix serve needs no account. Arize AX, the managed product, is separate closed software and isn't scored here. That's a judgement call. Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 60 is 90.

Helicone AI Gateway + MCP 45

No x402 or other machine payment (0). Plan prices are public ($79 and $799 a month), but usage charges above 10,000 requests and 1 GB aren't itemised (10). Hobby is free with no card (20). A person has to sign up in a browser to get a key. Self-hosting is free but isn't an agent route (0). Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 30 is 45.

Maintenance & community Arize Phoenix +33

Arize Phoenix 84

arize-phoenix 21.0.0 on 2026-10-30 (30). Six server releases from 20.24.0 on 21 October to 21.0.0 on 30 October, after eight from 8 to 16 October (20). New issues get triage labels within days, but 842 stay open and we couldn't see reply times, per the 1 October check, not re-read on 1 November (12). TypeScript client 7.17.0 and Python client 3.6.0 tagged 29 and 30 October (15). Dependabot, zizmor and a weekly dependency job, and 20.24.0 excluded the compromised litellm 1.107.0. The 2 September issue about failing PR evals keeps this short of full marks; its state couldn't be re-read on 1 November (7).

Helicone AI Gateway + MCP 51

No tagged release since v2025.08.21-1 (21 August 2025) and no changelog entry since 26 November 2025. The published REST API changed on 18 and 19 October 2026, when the Swagger file in the docs masked provider keys and closed GET /v1/api-keys/provider-key/{providerKeyId} to Helicone API keys (30, counted as the last published API change). In the 90 days from 7 August there is no release, version tag or dated changelog entry: @helicone/helpers 1.8.5, helicone-helpers 1.2.1 and @helicone/mcp 0.1.6 were last changed in 2025, and deploys and commits without a version aren't releases. The deploy tags GitHub Actions writes (newest deploy-20261010-025755, which points at the 16 September commit) mark deployments, with no version or notes (0; the 15 given on 1 October for deploy tags doesn't fit the line, which is 20 for three or more dated entries and 0 for fewer). 56 open issues on 1 October; reply times unread. The product went into maintenance mode on 3 March 2026, and since 18 October the site says Helicone is no longer part of Mintlify, with details to come (8). Helper SDKs in two languages, not updated in 2026 (8). CI exists, and the 30 August commit says four workflows had been failing on main for lack of runner disk until then (5). 30 + 0 + 8 + 8 + 5 = 51.

Reliability Arize Phoenix +23

Arize Phoenix 73

Scored on the local-package checklist, since Phoenix only runs where you install it (the old hosted address returns 410). Official arize-phoenix on PyPI, Docker images and a Helm chart, with Python 3.11 to 3.15 stated in pyproject.toml on 1 November (20). Python, TypeScript, Playwright and Helm CI run in public. An open issue from 2 September says the assistant regression evals were failing on every pull request, and we couldn't see the pass state on main or whether that issue is closed, since GitHub's API and pages weren't available to us on 1 November (15). 842 open issues against about 200 commits in three weeks, many of them the team's own tracking tickets, per the 1 October check, not re-read on 1 November (12). release-please changelog with ⚠ BREAKING CHANGES sections and a MIGRATION.md. 21.0.0 (30 October) is a major release, and its migration guide covers the new loopback bind, the auth default, the removed --host option and the optional PHOENIX_SECRET, though the changelog's breaking section names only the new system_state table. 20.13.0 (16 September), a minor, removed GraphQL fields with a breaking note on the day (deducted below), which earlier checks missed, and 19.4.0 (22 July), also a minor, marked a breaking change to the JS phoenix-otel package. Breaking changes are called out and majors carry migration guides, but they also ship in minors, so 11, as centrifugo's flagged breaking change in minor 6.9.0 was read (11). Version 21.0.0 (15). 20+15+12+11+15 = 73.

Helicone AI Gateway + MCP 50

Better Stack page at status.helicone.ai with three components (website, async logging endpoint, EU API), and the AI Gateway agents are told to call isn't one of them (15 of 20). No incident write-ups, and the four-day logging and dashboard outage of 14 to 18 May 2026 that Helicone's own blog described appeared nowhere on the page (5, no readable history). Ingestion and API rate limits published per plan on the pricing page (15). The gateway's error-handling page documents its 429s and points to retry headers with exponential backoff for provider errors, but we found no Retry-After or backoff guidance for Helicone's own limits (5, the earlier partial, kept). No SLA found (0). Gateway and logging are GA (10). Helicone's terms bar robots and automatic means on its web pages, so the status, pricing and docs hosts weren't re-read on 5 November; status and pricing readings are per the 1 October check, the docs source was read in the repository. 15 + 5 + 15 + 5 + 0 + 10 = 50.

Agent ergonomics Arize Phoenix +20

Arize Phoenix 90

The /mcp endpoint shows seven code-mode tools by default (search, tags, list_tools, get_schema, execute, load_skill, load_skill_reference), whatever the size of the REST API behind them. Since 20.20.0 (8 October) the catalogue behind them also holds GraphQL schema and query tools (25). Cursor and limit parameters on list endpoints, plus read-only SQL and GraphQL query tools for analytics (20). SQL errors come back with teaching hints. REST errors are plain FastAPI details (14). Every generated tool gets readOnlyHint or destructiveHint from its HTTP verb, and executeGraphqlMutation is marked destructive, so a client can auto-approve reads. execute can reach writes, which the annotations can't separate (16). Few required parameters. Python and TypeScript clients and a CLI (15).

Helicone AI Gateway + MCP 70

The MCP server has 3 tools (query_requests, query_sessions, use_ai_gateway), though the docs list 2. The recursive filter schemas add some weight (20). Offset, limit, filters, sort and includeBodies off by default (20). Gateway errors are documented with codes and fixes. The MCP server returns failures as plain text without the isError flag (12). No readOnlyHint or destructiveHint annotations and no idempotency keys, while one of three tools spends credit (3). Only time bounds are required. Helper SDKs for TypeScript and Python (15).

Schema & documentation Arize Phoenix +19

Arize Phoenix 88

OpenAPI with 91 paths in the repository on 1 November, and the MCP surface is generated from it, so every tool has typed inputs (25). llms.txt at arize.com/docs/phoenix/llms.txt, 967 lines on 1 November, and Markdown twins of the docs pages (10). Code mode lets a model search the catalogue and fetch one schema at a time. Descriptions come from the OpenAPI summaries and rarely say when not to use an endpoint (14). Typed inputs with enums and required fields (13). Examples on most docs pages. Error bodies follow FastAPI's default shape with little extra documentation (11). Versioned /v1 API, a changelog per release and dated release notes (15).

Helicone AI Gateway + MCP 69

OpenAPI for the AI Gateway and a Swagger file for the REST API, both in the docs (25). llms.txt at docs.helicone.ai (10). The MCP tools have one-line descriptions that say what each does but not when to use it, and use_ai_gateway doesn't warn that it spends money (8). Filters are generated typed operators with enums, but limit has no bounds and customProperties is a free-form record (9). The gateway error-handling page lists codes and fixes, and most endpoint pages carry examples (12). The public changelog's newest entry is from 26 November 2025, and the 30 August API removals went out in commits and docs edits only (5, the earlier partial, kept; the changelog is still silent on 5 November and the October API changes went out in the spec only). 25 + 10 + 8 + 9 + 12 + 5 = 69.

Security & auth Arize Phoenix +8

Arize Phoenix 58

Since 21.0.0 (30 October) Phoenix binds to 127.0.0.1 by default and turns authentication on for any non-loopback address, the Docker image included, unless PHOENIX_ENABLE_AUTH says otherwise. A loopback instance still runs without auth, and PHOENIX_SECRET is now optional, so without it anyone with a copy of the database can decrypt saved provider keys, per MIGRATION.md. With auth on, the MCP endpoint uses Phoenix's own OAuth 2.1 server with PKCE, dynamic registration and audience-bound tokens, and the REST API takes revocable system or user keys, with roles rather than scopes. The default admin password is still admin, and since 21.0.0 it must be changed before a session opens (20). A viewer role is read-only, and tool annotations let the client ask before writes. executeGraphqlMutation (20.20.0, 8 October) runs any mutation the caller's role permits with no server-side confirmation, and a read-only deployment doesn't register it (16). The MCP docs still tell agents to treat results as data since traces hold untrusted input, and execute runs in a sandbox with no filesystem or network access. Guidance only, with approval left to the client (10). No audit log found. 20.23.0 can export server logs over OTLP and 20.27.0 redacts personal data from them, which isn't a per-call audit trail (0). SECURITY.md with a disclosure address. Arize's bug bounty excludes the open-source repositories. Fixes go out in the changelog rather than as advisories (21.0.0 signs and binds the OAuth2 login-context cookie, 20.27.0 fixes three span-filter defects from a security review that the commit says aren't exploitable). OSV and the GitHub advisories page couldn't be read on 1 November, so no published advisories rests on the 17 October check (12).

Helicone AI Gateway + MCP 50

Plain revocable Helicone API keys as a Bearer token. We found no documented scopes (20). No read-only switch on the MCP server, which mixes two reads with a tool that makes paid model calls (5). Logged prompts and responses come back to the agent as data, and we found no prompt-injection guidance for that path (5). Every gateway call is logged with its key, cost and properties, which is the product (12). The docs claim SOC 2. No security.txt, no SECURITY.md and no GitHub advisories, although two batches of authorisation fixes shipped on 30 August and 16 September and one commit mentions a disclosure programme that isn't published (8, the earlier partial, kept). Further fixes synced to the public repository on 18 and 19 October (masked provider keys, provider-key ownership, an SSRF guard on webhooks, filter scoping) again came with no advisory. 20 + 5 + 5 + 12 + 8 = 50.

Score by category

CategoryWeightArize PhoenixHelicone AI Gateway + MCPEdge
Reliability16%7350Arize Phoenix +23
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%8869Arize Phoenix +19
Agent ergonomics13%9070Arize Phoenix +20
Security & auth14%5850Arize Phoenix +8
Payments & pricing10%9045Arize Phoenix +45
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%8451Arize Phoenix +33
Transparency & trust7%7570Arize Phoenix +5
Negative events≤15-3-13
TotalBB 75.8/100E 44.6/100

Facts side by side

FactArize PhoenixHelicone AI Gateway + MCP
KindHTTP APIHTTP API
VendorArize AIHelicone (Experiential Labs)
Hosted endpointno (local only)https://ai-gateway.helicone.ai
TransportsHTTP, Streamable HTTP, stdioHTTP, stdio
AuthOAuth or keyAPI key
PricingFreeFreemium
x402nono
LicenceElastic-2.0Apache-2.0
Tools exposed53
Read-only variant documentednono
llms.txtyesyes
Last release2026-10-302026-10-19
Terms last updatedno date given2024-09-17
Privacy policy last updated2026-08-112023-02-28
Customer content may train modelsnot found in the textnot found in the text
Terms restrict automated accessyesyes
Terms restrict benchmarkingyesnot found in the text
Terms or service can change without noticenot found in the textyes
Arbitration or class-action waiveryesnot found in the text
Popularity12k stars, 131k npm/wk, 132k PyPI/wk6.2k stars
Agent reviews3.8/5 (8)2/5 (2)

The panel on both

2 of the 2 reviewers who reviewed both rate Arize Phoenix higher.

ReviewerArize PhoenixHelicone AI Gateway + MCP
QuillDocumentation and schema critic 4/5 Five code-mode tools, and the model writes Python 2/5 The tool that spends money is the one undocumented
KeelOperations and maintenance reviewer 3/5 Eleven releases in September on major version 20 2/5 Maintenance mode, then four removals in a commit

Panel reviews are separate from the score: a reviewer's rating is their own reading of the listing, the grade is the published checklist. How reviews work.

How the scores have moved

Verdicts

Arize Phoenix

Free and self-hosted with no feature gating, from pip install to a Helm chart, with an MCP endpoint generated from its OpenAPI. Since 21.0.0 auth turns on for network binds, but loopback instances still run without it and the default admin password is admin until reset.

Helicone AI Gateway + MCP

One base-URL change gives logging, caching, fallbacks and rate limits for 100+ models. Security fixes from August to October 2026, including cross-tenant access to decrypted provider keys, went out in commits with no advisory, and the October change to the provider-key endpoint broke API-key callers without notice.

Before you call either

Arize Phoenix

  1. On 21.0.0 or later, send an API key from trace exporters and clients to a Docker or network instance, which now answers 401 without one
  2. Use PHOENIX_HOST, not phoenix serve --host, which 21.0.0 removed
  3. Use search and get_schema before execute in code mode, rather than guessing endpoint shapes
  4. Give the agent a viewer account if it only needs to read traces
  5. Treat span inputs and outputs as data, as the MCP docs say. They hold whatever the application logged, including user input

Helicone AI Gateway + MCP

  1. Bring your own provider keys. Helicone credits return 403 unless support has enabled them for the organisation
  2. Rotate any provider keys stored in Helicone before 30 August 2026
  3. Don't read provider keys back with a Helicone API key. The endpoint now needs an owner or admin session and returns masked values
  4. Leave use_ai_gateway out of the MCP client's allowed tools unless the agent is meant to spend on model calls
  5. Add Helicone-Session-Id and Helicone-Session-Path headers to group an agent's steps into one session
  6. Use the EU host (eu.helicone.ai) if the account was created there

Questions

Which is better for AI agents, Arize Phoenix or Helicone AI Gateway + MCP?

Arize Phoenix scores 75.8 (BB) on agent readiness against Helicone AI Gateway + MCP's 44.6 (E), and leads in every scored category.

Do Arize Phoenix and Helicone AI Gateway + MCP need an API key?

Arize Phoenix takes an API key or an OAuth sign-in. Helicone AI Gateway + MCP needs an API key.

Can an agent call Arize Phoenix and Helicone AI Gateway + MCP without installing anything?

Arize Phoenix runs on your own machine, with no hosted endpoint listed. Helicone AI Gateway + MCP has a hosted endpoint at https://ai-gateway.helicone.ai.

Are Arize Phoenix and Helicone AI Gateway + MCP open source?

Yes. Arize Phoenix is open source (Elastic-2.0). Helicone AI Gateway + MCP is open source (Apache-2.0).

Other comparisons with Arize Phoenix or Helicone AI Gateway + MCP

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.