Arize Phoenix vs DeepEval

Arize Phoenix

Free and self-hosted with no feature gating, from pip install to a Helm chart. Auth is off by default and the default admin password is admin.

DeepEval

DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. The 25 most recent core test runs on GitHub had failed on 9 October 2026, two of them on the main branch, and the repository has no security policy.

Arize Phoenix

1. Turn on auth before exposing the server, and change the admin password
2. Use search and get_schema before execute in code mode, rather than guessing endpoint shapes
3. Give the agent a viewer account if it only needs to read traces
4. Treat span inputs and outputs as data. They hold whatever the application logged
5. Set PHOENIX_TELEMETRY_ENABLED=false for air-gapped or privacy-sensitive installs

DeepEval

1. Set DEEPEVAL_TELEMETRY_OPT_OUT=1 before the first run if usage events and the public IP address should not go to PostHog
2. Set a judge model key such as OPENAI_API_KEY, or use the non-LLM metrics. Most metrics call an LLM judge and bill the owner's provider account
3. Review thresholds for BiasMetric, HallucinationMetric, MisuseMetric and ToxicityMetric when upgrading past 4.2.0. Higher scores now mean better
4. Read results from .deepeval/.latest_run_full.json or a results_folder. deepeval inspect opens a terminal interface meant for a person
5. Pass an existing key with deepeval login --api-key in CI. Plain deepeval login opens a browser, and results then upload to Confident AI