DeepEval vs Prefactor

DeepEval

DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. The 25 most recent core test runs on GitHub had failed on 9 October 2026, two of them on the main branch, and the repository has no security policy.

Prefactor

The public OpenAPI document lists 157 operations, each accepting an idempotency key, and tokens can be read-only or bound to one agent deployment. The pricing and security pages describe approval, blocking and PII detection that the documentation says the product does not do, and the only published terms cover the website.

DeepEval

1. Set DEEPEVAL_TELEMETRY_OPT_OUT=1 before the first run if usage events and the public IP address should not go to PostHog
2. Set a judge model key such as OPENAI_API_KEY, or use the non-LLM metrics. Most metrics call an LLM judge and bill the owner's provider account
3. Review thresholds for BiasMetric, HallucinationMetric, MisuseMetric and ToxicityMetric when upgrading past 4.2.0. Higher scores now mean better
4. Read results from .deepeval/.latest_run_full.json or a results_folder. deepeval inspect opens a terminal interface meant for a person
5. Pass an existing key with deepeval login --api-key in CI. Plain deepeval login opens a browser, and results then upload to Confident AI

Prefactor

1. Send requests to https://app.prefactorai.com/api/v1 with Authorization: Bearer <token>. The API host is not on the prefactor.tech or prefactor.ai domains
2. Ask for a deployment-scoped token, or an account token created with role set to read_only, since the default role is full_access with a two-year expiry
3. Pass redacted: true on span list and detail queries, because the API returns marked sensitive values unless asked otherwise
4. On 429 wait for retry_after_ms. Limits use one-minute windows and the SDKs' default retries ignore the server's hint
5. Treat terminate as a request. Check the control signal on span responses or poll the instance, because Prefactor does not stop the process