DeepEval vs LangWatch

DeepEval

DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. The 25 most recent core test runs on GitHub had failed on 9 October 2026, two of them on the main branch, and the repository has no security policy.

LangWatch

API keys can be limited to read or write per permission category, expire, and be revoked, and the remote MCP server uses OAuth with PKCE. The MCP server registers 101 tools with no read-only or destructive annotations, and no rate limit for the platform API was found in the reviewed documentation.

DeepEval

1. Set DEEPEVAL_TELEMETRY_OPT_OUT=1 before the first run if usage events and the public IP address should not go to PostHog
2. Set a judge model key such as OPENAI_API_KEY, or use the non-LLM metrics. Most metrics call an LLM judge and bill the owner's provider account
3. Review thresholds for BiasMetric, HallucinationMetric, MisuseMetric and ToxicityMetric when upgrading past 4.2.0. Higher scores now mean better
4. Read results from .deepeval/.latest_run_full.json or a results_folder. deepeval inspect opens a terminal interface meant for a person
5. Pass an existing key with deepeval login --api-key in CI. Plain deepeval login opens a browser, and results then upload to Confident AI

LangWatch

1. Create a Restricted key with read access to only the categories the task needs. A personal key with All permissions carries everything its owner can do
2. Set both LANGWATCH_API_KEY and LANGWATCH_PROJECT_ID for the MCP server unless the key reaches only one project
3. Call discover_schema before search_traces or get_analytics, and keep the default digest format. json returns the full raw trace
4. Allowlist MCP tools in the client. All 101 load by default, among them platform_create_api_key and the delete tools
5. Follow next_cursor until it is null on list endpoints. A full page does not mean more rows exist