DeepEval vs Respan API + MCP

DeepEval

DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. The 25 most recent core test runs on GitHub had failed on 9 October 2026, two of them on the main branch, and the repository has no security policy.

Respan API + MCP

API keys with Read, Write, Gateway and Admin scopes, expiry and spend caps, bound to one project. 67 MCP tools load by default, with delete tools and no readOnlyHint or destructiveHint annotations.

DeepEval

1. Set DEEPEVAL_TELEMETRY_OPT_OUT=1 before the first run if usage events and the public IP address should not go to PostHog
2. Set a judge model key such as OPENAI_API_KEY, or use the non-LLM metrics. Most metrics call an LLM judge and bill the owner's provider account
3. Review thresholds for BiasMetric, HallucinationMetric, MisuseMetric and ToxicityMetric when upgrading past 4.2.0. Higher scores now mean better
4. Read results from .deepeval/.latest_run_full.json or a results_folder. deepeval inspect opens a terminal interface meant for a person
5. Pass an existing key with deepeval login --api-key in CI. Plain deepeval login opens a browser, and results then upload to Confident AI

Respan API + MCP

1. Create a key with only the Read scope for agents that analyse logs. New keys get Read, Write and Gateway by default
2. Set Respan-Enabled-Tools to a short list such as list_traces,get_trace_tree,list_logs. All 67 tools load otherwise
3. Pass filters to list_logs rather than paging through everything. The default window is the last hour
4. Call list_organizations first when results look empty over OAuth. Every tool acts on the active organisation only
5. Append .md to any docs URL for clean Markdown