DeepEval vs LangSmith API + MCP

DeepEval

DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. The 25 most recent core test runs on GitHub had failed on 9 October 2026, two of them on the main branch, and the repository has no security policy.

LangSmith API + MCP

Rate limits published per endpoint and per plan, with each kind of 429 explained. Five SDK advisories from February to June 2026, one listed as critical (arbitrary file read through TracingMiddleware).

DeepEval

1. Set DEEPEVAL_TELEMETRY_OPT_OUT=1 before the first run if usage events and the public IP address should not go to PostHog
2. Set a judge model key such as OPENAI_API_KEY, or use the non-LLM metrics. Most metrics call an LLM judge and bill the owner's provider account
3. Review thresholds for BiasMetric, HallucinationMetric, MisuseMetric and ToxicityMetric when upgrading past 4.2.0. Higher scores now mean better
4. Read results from .deepeval/.latest_run_full.json or a results_folder. deepeval inspect opens a terminal interface meant for a person
5. Pass an existing key with deepeval login --api-key in CI. Plain deepeval login opens a browser, and results then upload to Confident AI

LangSmith API + MCP

1. Batch run reads. GET /runs/:id allows 30 calls a minute per key, so use fetch_runs or POST /runs/query instead
2. Pass trace_id or min_start_time to fetch_runs and keep limit low. Results page by a 25,000-character budget
3. Don't call create_dataset or run_experiment over MCP to make changes. They return how-to text, not results
4. Send X-Api-Key and point at the regional host (eu., apac. or aws.api.smith.langchain.com) that matches the workspace
5. Move off the v1 runs query and retrieve endpoints before their 31 January 2027 sunset