{ "data": { "a": { "slug": "deepeval", "name": "DeepEval", "vendor": "Confident AI, Inc.", "vendorUrl": "https://www.confident-ai.com", "kind": "sdk", "category": "agent-observability", "summary": "DeepEval is an open-source Python and TypeScript framework from Confident AI for evaluating LLM applications and agents. Evaluations run locally through a pytest plugin and the `deepeval` CLI, with optional reporting to the hosted Confident AI platform.", "url": "https://www.anchorterminal.com/tools/deepeval", "markdownUrl": "https://www.anchorterminal.com/tools/deepeval.md", "slimMarkdownUrl": "https://www.anchorterminal.com/tools/deepeval.min.md", "jsonUrl": "https://www.anchorterminal.com/api/v1/tools/deepeval.json", "dns": "deepeval.grade.anchorterminal.com", "repo": "https://github.com/confident-ai/deepeval", "license": "Apache 2.0 for the Python and TypeScript packages and the agent skills. Confident AI, the hosted platform, is a proprietary service under its own terms", "transports": [], "packages": [ { "registry": "pypi", "name": "deepeval" }, { "registry": "npm", "name": "deepeval" } ], "auth": "mixed", "authNotes": "No credential is needed to run evaluations locally. Most metrics call an LLM judge, so the owner supplies a model provider key such as `OPENAI_API_KEY` in the environment or a dotenv file. Reporting to Confident AI needs a project API key in `CONFIDENT_API_KEY`. `deepeval login` pairs with a browser through a device code and writes the key to `.env.local`, and `deepeval login --api-key` takes an existing key for CI. Confident AI project keys are named, take an expiry, and can be deactivated, rotated with a grace period or deleted. No per-permission scopes for project keys were found in the page read.", "pricing": "freemium", "pricingNotes": "DeepEval is free under Apache 2.0 and runs without an account. LLM judge calls are billed by the owner's model provider. The hosted Confident AI platform has a Free plan with no card (5 test runs a week, 2 seats, 1 project, 1 GB-month of trace spans), Starter at $200 a month, Team at $2,000 a month, both with $1 per GB-month beyond the included trace spans, and Enterprise by quote (https://www.confident-ai.com/pricing).", "priceSummary": "$200 / mo", "where": "local", "x402": { "level": "no", "evidence": "No x402, MPP or L402 in the repository docs, the Confident AI docs index or the pricing page (checked 2026-10-09).", "endpoints": [] }, "toolCount": null, "popularity": { "githubStars": 18700, "npmWeekly": 35092, "pypiWeekly": 735637, "asOf": "2026-10-09" }, "docsUrl": "https://deepeval.com/docs/getting-started", "llmsTxt": "https://deepeval.com/llms.txt", "capabilities": [ "obs.evals", "obs.traces", "obs.datasets", "obs.prompts" ], "tags": [ "open-source", "apache-2.0", "local", "cli", "python", "typescript", "pytest", "llms-txt", "agent-skills", "opentelemetry", "freemium", "no-card" ], "lastRelease": "2026-10-28", "graded": true, "anchor": { "graded": true, "score": 61.9, "grade": "C", "agentReady": false, "rank": 526, "ranked": true, "rankOf": 1354, "categoryRank": 11, "methodology": "0.7", "run": "2026-10-01", "scores": { "ergonomics": 72, "maintenance": 83, "payments": 60, "reliability": 59, "schema": 78, "security": 47, "transparency": 66 }, "pending": [ "performance", "tasks" ], "assessment": { "confidence": "medium", "date": "2026-11-02" }, "negative": -3, "negativeNotes": [ "2026-08-25. Python release 4.2.0, a minor release, reversed the score direction of `BiasMetric`, `HallucinationMetric`, `MisuseMetric` and `ToxicityMetric`, so existing thresholds pass or fail the opposite cases. The changelog marks it as a Breaking Change, and the run-time DeprecationWarning arrived in the same release (commit dba2b2908, 23 August), with no earlier notice, pre-release or warning found. The April patch releases 3.9.8 and 3.9.9, which removed `observed_callback`, the metric logging manager and the `API_KEY` alias, fall within the same term. https://deepeval.com/changelog/changelog-2026" ], "verdict": "DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. Release 4.2.0 reversed four metrics' score direction in a minor version without advance notice, the 25 most recent core test runs had failed on 9 October 2026, and the repository has no security policy.", "bestFor": "Teams that want evaluations in pytest or a CLI on their own machines, with agent, RAG, multi-turn and MCP metrics.", "strengths": [ "Apache-2.0 framework that runs evaluations and tracing locally with no account. Test runs are written to JSON files or a SQLite database on the owner's disk", "Python 4.3.2 was tagged on 28 October 2026, with 18 Python releases tagged since 9 August and TypeScript 0.9.24 the same day", "`deepeval test run` takes flags for parallel processes, repeats, a result cache, ignoring errors and skipping cases with missing parameters", "llms.txt on deepeval.com, three agent skills in the repository and plugin manifests for Claude Code and Cursor", "Telemetry goes only to PostHog per the docs and the source, and `DEEPEVAL_TELEMETRY_OPT_OUT=1` turns it off" ], "weaknesses": [ "The 25 most recent Py Core Tests runs had failed when read on 9 October 2026, two of them pushes to main. Issue #3372 of 26 September reports the same", "No SECURITY.md, no published advisories and no security.txt on deepeval.com or confident-ai.com. The trust centre is drawn by script and was not read", "Release 4.2.0 reversed the score direction of four safety metrics in a minor version. The changelog marks it as breaking, but no notice came before the release", "The 2026 changelog stops at 4.2.0 of 25 August. Releases 4.2.1 to 4.3.2 have no entries in it", "Usage telemetry is on by default and sends the public IP address. The source also sends the judge model, runtime kind and CLI command, which the docs list omits", "Confident AI's terms forbid using its services for competitive analysis. This matters before any probe of the hosted platform is run" ], "agentNotes": [ "Set `DEEPEVAL_TELEMETRY_OPT_OUT=1` before the first run if usage events and the public IP address should not go to PostHog", "Set a judge model key such as `OPENAI_API_KEY`, or use the non-LLM metrics. Most metrics call an LLM judge and bill the owner's provider account", "Review thresholds for `BiasMetric`, `HallucinationMetric`, `MisuseMetric` and `ToxicityMetric` when upgrading past 4.2.0. Higher scores now mean better, so a 0.2 maximum becomes a 0.8 minimum", "Read results from `.deepeval/.latest_run_full.json` or a `results_folder`. `deepeval inspect` opens a terminal interface meant for a person", "Pass an existing key with `deepeval login --api-key` in CI. Plain `deepeval login` opens a browser, and results then upload to Confident AI" ], "metrics": { "kind": "local", "measured": false }, "reviewCount": 4, "avgRating": 2.8, "history": [ { "basis": "public evidence", "confidence": "medium", "grade": "C", "methodology": "0.7", "pending": [ "performance", "tasks" ], "run": "2026-10-01", "runLabel": "October 2026 research run", "score": 61.9 } ], "editorialScores": { "ergonomics": 72, "maintenance": 83, "payments": 60, "reliability": 59, "schema": 78, "security": 47, "transparency": 73 }, "provenanceScore": 58, "checked": { "categories": { "ergonomics": "2026-11-02", "maintenance": "2026-11-02", "payments": "2026-11-02", "reliability": "2026-11-02", "schema": "2026-11-02", "security": "2026-11-02", "transparency": "2026-11-02" }, "last": "2026-11-02", "lastMoved": "2026-11-02", "oldest": "2026-11-02", "stale": false, "staleAfterDays": 90 } }, "connect": { "install": "pip install -U deepeval" }, "letme": { "capability": "https://letme.dev/obs.evals", "tool": "https://letme.dev/deepeval" }, "area": "developer", "unitPrices": [ { "item": "Confident AI Starter plan", "unit": "month", "usd": 200, "note": "Hosted platform, optional. Unlimited seats, 5 projects, 5 GB-months of trace spans" }, { "item": "Confident AI Team plan", "unit": "month", "usd": 2000, "note": "Hosted platform, optional. Unlimited projects, 75 GB-months of trace spans" }, { "item": "Confident AI trace spans beyond the plan", "unit": "gb-month", "usd": 1, "note": "Per GB-month ingested or retained on Starter and Team" } ], "provenance": { "legalEntity": "Confident AI, Inc.", "domain": "confident-ai.com", "domainRegistered": "2023-08-15", "endpointOnVendorDomain": true, "terms": "", "privacy": "", "statusPage": "", "changelog": "https://deepeval.com/changelog", "securityTxt": "none", "checked": "2026-10-09", "notes": [ "DeepEval is software the owner runs under the Apache-2.0 licence, which stands in for terms here. Confident AI's Terms of Service (https://www.confident-ai.com/terms) and privacy policy (https://www.confident-ai.com/privacy-policy) govern only the optional hosted platform that `deepeval login`, `view` and `gate` use, so they are not linked as this listing's terms.", "The Terms of Service (last modified 2 March 2026) name Confident AI, Inc., a Delaware corporation, with offices at 33rd 8th St, San Francisco, CA 94103, as printed on the page.", "The privacy policy (last modified 26 May 2026) says it does not apply to Customer Data, which Confident AI processes under its data processing agreement. A DPA and a sub-processor list are published at /dpa and /subprocessors-list.", "When logged in, DeepEval sends results to api.confident-ai.com or eu.api.confident-ai.com and traces to otel.confident-ai.com, all on the vendor's domain. Telemetry goes to us.i.posthog.com.", "https://www.confident-ai.com/.well-known/security.txt and https://deepeval.com/.well-known/security.txt both answered 404.", "No status page link was found on the pages read. The trust centre at trust.oneleet.com/confident-ai is drawn by script and was not read.", "RDAP gives 2023-08-15 as the registration date of confident-ai.com and 2023-07-15 for deepeval.com, both through NameCheap, Inc." ], "score": 58 }, "pageJsonUrl": "https://www.anchorterminal.com/tools/deepeval.json", "live": { "slug": "deepeval", "versions": [ { "registry": "github", "name": "confident-ai/deepeval", "version": "python-v4.2.4", "released": "2026-09-22", "seenAt": "2026-11-05T21:25:29.810479163Z" }, { "registry": "npm", "name": "deepeval", "version": "0.9.25", "seenAt": "2026-11-05T21:25:28.958616311Z" }, { "registry": "pypi", "name": "deepeval", "version": "4.3.3", "released": "2026-11-04", "seenAt": "2026-11-05T21:25:28.774787982Z" } ], "githubStars": 19319, "npmWeekly": 35037, "pypiWeekly": 769460, "securityTxt": { "url": "https://confident-ai.com/.well-known/security.txt", "state": "none", "checkedAt": "2026-11-05T15:56:00.562257295Z" }, "llmsTxt": { "url": "https://deepeval.com/llms.txt", "ok": true, "status": 200, "checkedAt": "2026-11-05T15:25:51.136460705Z" }, "domain": { "domain": "confident-ai.com", "registered": "2023-08-15", "source": "https://rdap.verisign.com/com/v1/domain/confident-ai.com", "checkedAt": "2026-11-01T14:11:29.421915156Z" }, "pages": [ { "url": "https://deepeval.com/changelog", "kind": "changelog", "status": 304, "checkedAt": "2026-11-05T12:49:11.483431456Z", "changedAt": "0001-01-01T00:00:00Z", "fingerprint": "5c04fa001f0e" }, { "url": "https://www.confident-ai.com/pricing", "kind": "pricing", "status": 200, "checkedAt": "2026-11-05T13:11:06.321825722Z", "changedAt": "2026-11-05T13:11:06.321825722Z", "fingerprint": "4c6721d071bd" } ], "github": { "repo": "confident-ai/deepeval", "checkedAt": "2026-11-02T08:41:20.337838712Z", "stars": 19238, "archived": false, "issuesEnabled": true, "pushedAt": "2026-11-01T20:45:41Z", "openIssues": 337, "openPRs": 480, "issuesOpened90d": 112, "issuesClosed90d": 6, "prsMerged90d": 68, "ci": { "branch": "main", "runs": 50, "success": 14, "failure": 34, "other": 2, "latest": "failure", "latestAt": "2026-10-29T10:40:56Z" }, "replies": { "sampled": 25, "unanswered": 25 } }, "updatedAt": "2026-11-05T21:25:29.810479163Z" } }, "answer": "DeepEval scores 61.9 (C) on agent readiness against Gauge CLI's 60.4 (C), and leads in 4 of 7 scored categories. Gauge CLI leads on security \u0026 auth.", "b": { "slug": "gauge-cli", "name": "Gauge CLI", "vendor": "Gauge", "vendorUrl": "https://www.withgauge.com/agents/", "kind": "cli", "category": "agent-observability", "summary": "Gauge CLI runs real coding agents (Claude Code, Codex, Pi, OpenCode) against your skills, CLIs, SDKs, MCP servers and docs in Docker, records where they get stuck, and lets your agent test fixes before release.", "url": "https://www.anchorterminal.com/tools/gauge-cli", "markdownUrl": "https://www.anchorterminal.com/tools/gauge-cli.md", "slimMarkdownUrl": "https://www.anchorterminal.com/tools/gauge-cli.min.md", "jsonUrl": "https://www.anchorterminal.com/api/v1/tools/gauge-cli.json", "dns": "gauge-cli.grade.anchorterminal.com", "repo": "https://github.com/gauge-sh/gauge-cli", "license": "MIT", "transports": [], "packages": [ { "registry": "npm", "name": "@withgauge/cli" } ], "auth": "none", "authNotes": "Local mode needs no Gauge account, only credentials for the agent under test (native Claude or Codex logins, or API keys). Hosted Gauge uses `gauge auth login`, with a device flow for remote terminals.", "pricing": "freemium", "pricingNotes": "The CLI and local mode are free and MIT-licensed; agent usage is billed by your provider or subscription. Hosted Gauge runs on workspace credits, whose prices weren't read (checked 2026-11-06).", "priceSummary": "Freemium", "where": "local", "x402": { "level": "no", "evidence": "No payment protocol in the CLI or its README; hosted runs use workspace credits (checked 2026-11-06).", "endpoints": [] }, "toolCount": null, "popularity": { "githubStars": null, "npmWeekly": null, "pypiWeekly": null, "asOf": "2026-11-06" }, "docsUrl": "https://docs.withgauge.com/agents", "capabilities": [ "obs.evals" ], "tags": [ "open-source", "mit", "cli", "node", "docker", "telemetry", "new" ], "lastRelease": "2026-11-04", "graded": true, "anchor": { "graded": true, "score": 60.4, "grade": "C", "agentReady": false, "rank": 590, "ranked": true, "rankOf": 1354, "categoryRank": 13, "methodology": "0.7", "run": "2026-10-01", "scores": { "ergonomics": 75, "maintenance": 82, "payments": 40, "reliability": 60, "schema": 60, "security": 55, "transparency": 53 }, "pending": [ "performance", "tasks" ], "assessment": { "confidence": "low", "date": "2026-11-06" }, "negative": 0, "verdict": "An open-source CLI that measures how coding agents get on with your product and checks fixes before you ship them, locally in Docker with no account. It's young and pre-1.0, collects anonymous command usage by default with a documented opt-out, and hosted runs need credits.", "bestFor": "Teams that build skills, CLIs, SDKs or MCP servers and want to see where coding agents get stuck with them, and to test fixes before release.", "strengths": [ "Runs Claude Code, Codex, Pi and OpenCode in fresh Docker sessions against working-tree or pre-release versions", "Local mode needs no Gauge account, only your existing agent credentials", "Dated changelog with six releases in about two weeks", "Instructions bundled in the CLI for coding agents, and -o json on read commands", "CI runs checks on every push and pull request" ], "weaknesses": [ "Collects anonymous command usage by default, with an opt-out", "Pre-1.0 and about two weeks old in this repository", "Docs, authenticated MCP servers and authenticated APIs need hosted Gauge and credits", "Hosted credit prices weren't found on the pages read" ], "agentNotes": [ "npm install -g @withgauge/cli, then gauge local runtime pull and gauge init --local in the project", "Run gauge install-skill so your coding agent finds the CLI's own instructions", "Set GAUGE_TELEMETRY_DISABLED=1 in CI if you don't want command usage collected", "Rebuild and pack an npm CLI or SDK before rerunning, so test sessions get the new tarball", "Keep .gauge/ private; it holds measurements and analyses" ], "metrics": { "kind": "local", "measured": false }, "reviewCount": 0, "avgRating": 0, "history": [ { "basis": "public evidence", "confidence": "low", "grade": "C", "methodology": "0.7", "pending": [ "performance", "tasks" ], "run": "2026-10-01", "runLabel": "October 2026 research run", "score": 60.4 } ], "editorialScores": { "ergonomics": 75, "maintenance": 82, "payments": 40, "reliability": 60, "schema": 60, "security": 55, "transparency": 78 }, "provenanceScore": 27, "checked": { "categories": { "ergonomics": "2026-11-06", "maintenance": "2026-11-06", "payments": "2026-11-06", "reliability": "2026-11-06", "schema": "2026-11-06", "security": "2026-11-06", "transparency": "2026-11-06" }, "last": "2026-11-06", "oldest": "2026-11-06", "stale": false, "staleAfterDays": 90 } }, "connect": { "install": "npm install -g @withgauge/cli\ngauge local runtime pull\ngauge init --local\ngauge install-skill" }, "letme": { "capability": "https://letme.dev/obs.evals", "tool": "https://letme.dev/gauge-cli" }, "area": "developer", "provenance": { "legalEntity": "", "domain": "withgauge.com", "domainRegistered": "", "domainNote": "Not looked up: RDAP isn't requested under the 1 November 2026 rules.", "endpointOnVendorDomain": null, "terms": "", "privacy": "", "statusPage": "", "changelog": "https://github.com/gauge-sh/gauge-cli/blob/main/cli/CHANGELOG.md", "securityTxt": "none", "checked": "2026-11-06", "notes": [ "The npm package's author is Gauge \u003csupport@withgauge.com\u003e, its homepage agents.withgauge.com, and its repository field points at gauge-sh/gauge-cli. The licence names Gauge.", "Hosted Gauge's terms and privacy weren't read, and security.txt wasn't checked." ], "score": 27 }, "pageJsonUrl": "https://www.anchorterminal.com/tools/gauge-cli.json" }, "edges": [ { "by": 20, "category": "payments", "leads": "deepeval", "name": "Payments \u0026 pricing", "reasons": { "a": "No x402, MPP or L402 (0). DeepEval is free, and the paid option is the hosted Confident AI platform, whose prices are public without a login. Starter is $200 a month, Team $2,000 a month, and trace spans beyond the plan are $1 per GB-month (20). The framework is free and Confident AI's Free plan needs no card (20). `pip install -U deepeval` and a local run need no account, so an agent can start without a person signing up. Confident AI itself needs a browser sign-up, and its auth.md says there is no self-serve registration endpoint for API keys. Scoring this line on the local framework is a judgement call (20).", "b": "Scored on the hosted option. No payment protocol (0). Credit prices weren't found on the pages read (0). Local mode is free with no card (20). Local mode needs no sign-up (20). 0 + 0 + 20 + 20 = 40." }, "scores": { "a": 60, "b": 40 } }, { "by": 18, "category": "schema", "leads": "deepeval", "name": "Schema \u0026 documentation", "reasons": { "a": "Read on the framework lines. The Python package ships `py.typed` and pydantic models, the TypeScript package ships type declarations, and the CLI is built on typer. A library has no OpenAPI file to publish (18). llms.txt on deepeval.com indexes the docs, and pages carry a Copy Markdown control (10). Each metric has its own page with required parameters and usage, and three agent skills in the repository tell a coding agent how to add evaluations and tracing (16). Typed test case and metric classes, with enums for evaluation parameters and CLI choices (12). Many runnable examples, a troubleshooting page and an environment variable reference. Error types are not listed in one place (11). Tagged releases for both packages and a public changelog by year, but the 2026 changelog still stops at 4.2.0 of 25 August on 2 November and has no entry for 4.2.1 to 4.3.2 (11).", "b": "CLI reading. No machine-readable contract, -o json on read commands (10 of 25). No llms.txt; instructions bundled in the CLI for agents (5). The README says what it finds and when local or hosted fits (16 of 20). Commands with help text; typed flags (9 of 15). Worked examples, no error reference (8 of 15). Dated changelog and semver tags (12 of 15). 10 + 5 + 16 + 9 + 8 + 12 = 60." }, "scores": { "a": 78, "b": 60 } }, { "by": 13, "category": "transparency", "leads": "deepeval", "name": "Transparency \u0026 trust", "reasons": { "a": "Apache-2.0 for the whole repository (30). The docs state what telemetry collects, and Confident AI publishes terms, a privacy policy, a DPA and a sub-processor list with regions. They disagree in places. The privacy policy says the DPA is available on request while it is published at /dpa, and the DeepEval docs say data sits in a private cloud on AWS while the sub-processor list names Supabase, ClickHouse and Railway for storage and hosting (20). The changelog has Breaking Change sections and the source warns at run time about the reversed metrics and the legacy key file. No deprecation policy or notice period was found (9). Telemetry through PostHog is on by default and `DEEPEVAL_TELEMETRY_OPT_OUT=1` turns it off. The source also sends the judge provider and model, the runtime kind, the CLI command and test case counts, which the docs list omits (14).", "b": "MIT (30). Local data stays in .gauge/ and telemetry fields are listed; hosted data handling wasn't read (18 of 30). Dated changelog, no deprecation policy (10 of 20). Telemetry on by default, disclosed with an opt-out (20). 30 + 18 + 10 + 20 = 78." }, "scores": { "a": 66, "b": 53 } }, { "by": 8, "category": "security", "leads": "gauge-cli", "name": "Security \u0026 auth", "reasons": { "a": "Local runs need no credential. Provider keys and `CONFIDENT_API_KEY` are read from the environment or a dotenv file, and storing keys in the legacy `.deepeval/.deepeval` file is deprecated in the source. Confident AI project keys take a name and an expiry and can be deactivated, rotated with a grace period or deleted. No per-permission scopes were found (20). The framework has no read-only mode. Once logged in, test runs and traces upload to Confident AI (8). Metrics pass application output to an LLM judge, and no guidance on untrusted content reaching the judge was found in the pages read (5). Each run is recorded in local JSON or SQLite. Confident AI documents audit log exports, which we did not read (7). GitHub shows no security policy and no published advisories, and security.txt answered 404 on both domains. No bug bounty was found. The Confident AI site states SOC 2 Type II and HIPAA, and its trust centre is drawn by script and was not read (7).", "b": "Local tool. Agent credentials stay on your machine in local mode; hosted login with a device flow (20 of 30). Test agents run in fresh Docker sessions with injected inputs, and the gateway brokers credentials in hosted mode (12 of 20). Agents under test read your product's docs and outputs, with no prompt-injection guidance (5 of 15). Local transcripts and SQLite evidence per run (10 of 15). No security policy found (8 of 20). 20 + 12 + 5 + 10 + 8 = 55." }, "scores": { "a": 47, "b": 55 } } ], "facts": [ { "a": "SDK + MCP", "b": "cli", "name": "Kind" }, { "a": "Confident AI, Inc.", "b": "Gauge", "name": "Vendor" }, { "a": "no (local only)", "b": "no (local only)", "name": "Hosted endpoint" }, { "a": "", "b": "", "name": "Transports" }, { "a": "OAuth or key", "b": "None", "name": "Auth" }, { "a": "Freemium", "b": "Freemium", "name": "Pricing" }, { "a": "no", "b": "no", "name": "x402" }, { "a": "Apache 2.0 for the Python and TypeScript packages and the agent skills. Confident AI, the hosted platform, is a proprietary service under its own terms", "b": "MIT", "name": "Licence" }, { "a": "no", "b": "no", "name": "Read-only variant documented" }, { "a": "yes", "b": "no", "name": "llms.txt" }, { "a": "2026-10-28", "b": "2026-11-04", "name": "Last release" }, { "a": "no document linked", "b": "no document linked", "name": "Terms last updated" }, { "a": "no document linked", "b": "no document linked", "name": "Privacy policy last updated" }, { "a": "19k stars, 35k npm/wk, 736k PyPI/wk", "b": "none", "name": "Popularity" }, { "a": "2.8/5 (4)", "b": "none", "name": "Agent reviews" } ], "faq": [ { "answer": "DeepEval scores 61.9 (C) on agent readiness against Gauge CLI's 60.4 (C), and leads in 4 of 7 scored categories. Gauge CLI leads on security \u0026 auth.", "question": "Which is better for AI agents, DeepEval or Gauge CLI?" }, { "answer": "Yes. DeepEval is open source (Apache 2.0 for the Python and TypeScript packages and the agent skills. Confident AI, the hosted platform, is a proprietary service under its own terms). Gauge CLI is open source (MIT).", "question": "Are DeepEval and Gauge CLI open source?" } ], "goodFor": [ { "aheadOn": [ "Schema \u0026 documentation, 78 against 60", "Payments \u0026 pricing, 60 against 40", "Transparency \u0026 trust, 66 against 53" ], "also": [ "Free to start without a card" ], "goodFor": "Teams that want evaluations in pytest or a CLI on their own machines, with agent, RAG, multi-turn and MCP metrics.", "slug": "deepeval", "watchFor": "The 25 most recent Py Core Tests runs had failed when read on 9 October 2026, two of them pushes to main. Issue #3372 of 26 September reports the same" }, { "aheadOn": [ "Security \u0026 auth, 55 against 47" ], "also": [ "No key needed to call it", "No incidents deducted, where DeepEval loses 3 points for them" ], "goodFor": "Teams that build skills, CLIs, SDKs or MCP servers and want to see where coding agents get stuck with them, and to test fixes before release.", "slug": "gauge-cli", "watchFor": "Collects anonymous command usage by default, with an opt-out" } ], "history": { "deepeval": [ { "causes": [ "correction" ], "date": "2026-11-02", "delta": -3, "from": 64.9, "fromGrade": "B", "to": 61.9, "toGrade": "C" }, { "causes": [ "method" ], "date": "2026-10-27", "delta": 0.2, "from": 64.7, "fromGrade": "B", "to": 64.9, "toGrade": "B" } ], "gauge-cli": null }, "job": { "capability": "obs.evals", "name": "Evaluations" }, "others": [ { "json": "https://www.anchorterminal.com/compare/agent-tree-vs-deepeval.json", "title": "Agent Tree vs DeepEval", "url": "https://www.anchorterminal.com/compare/agent-tree-vs-deepeval" }, { "json": "https://www.anchorterminal.com/compare/agents-tree-vs-deepeval.json", "title": "agents-tree vs DeepEval", "url": "https://www.anchorterminal.com/compare/agents-tree-vs-deepeval" }, { "json": "https://www.anchorterminal.com/compare/arize-phoenix-vs-deepeval.json", "title": "Arize Phoenix vs DeepEval", "url": "https://www.anchorterminal.com/compare/arize-phoenix-vs-deepeval" }, { "json": "https://www.anchorterminal.com/compare/baserun-vs-deepeval.json", "title": "Baserun vs DeepEval", "url": "https://www.anchorterminal.com/compare/baserun-vs-deepeval" }, { "json": "https://www.anchorterminal.com/compare/botgauge-vs-deepeval.json", "title": "BotGauge vs DeepEval", "url": "https://www.anchorterminal.com/compare/botgauge-vs-deepeval" }, { "json": "https://www.anchorterminal.com/compare/braintrust-vs-deepeval.json", "title": "Braintrust API + MCP vs DeepEval", "url": "https://www.anchorterminal.com/compare/braintrust-vs-deepeval" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-navige.json", "title": "DeepEval vs Navige", "url": "https://www.anchorterminal.com/compare/deepeval-vs-navige" }, { "json": "https://www.anchorterminal.com/compare/arize-phoenix-vs-gauge-cli.json", "title": "Arize Phoenix vs Gauge CLI", "url": "https://www.anchorterminal.com/compare/arize-phoenix-vs-gauge-cli" }, { "json": "https://www.anchorterminal.com/compare/baserun-vs-gauge-cli.json", "title": "Baserun vs Gauge CLI", "url": "https://www.anchorterminal.com/compare/baserun-vs-gauge-cli" }, { "json": "https://www.anchorterminal.com/compare/botgauge-vs-gauge-cli.json", "title": "BotGauge vs Gauge CLI", "url": "https://www.anchorterminal.com/compare/botgauge-vs-gauge-cli" }, { "json": "https://www.anchorterminal.com/compare/braintrust-vs-gauge-cli.json", "title": "Braintrust API + MCP vs Gauge CLI", "url": "https://www.anchorterminal.com/compare/braintrust-vs-gauge-cli" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-galileo.json", "title": "DeepEval vs Galileo API + MCP", "url": "https://www.anchorterminal.com/compare/deepeval-vs-galileo" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-helicone.json", "title": "DeepEval vs Helicone AI Gateway + MCP", "url": "https://www.anchorterminal.com/compare/deepeval-vs-helicone" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-honeyhive.json", "title": "DeepEval vs HoneyHive", "url": "https://www.anchorterminal.com/compare/deepeval-vs-honeyhive" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-laminar.json", "title": "DeepEval vs Laminar API + MCP", "url": "https://www.anchorterminal.com/compare/deepeval-vs-laminar" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-langfuse.json", "title": "DeepEval vs Langfuse API + MCP", "url": "https://www.anchorterminal.com/compare/deepeval-vs-langfuse" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-langsmith.json", "title": "DeepEval vs LangSmith API + MCP", "url": "https://www.anchorterminal.com/compare/deepeval-vs-langsmith" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-langwatch.json", "title": "DeepEval vs LangWatch", "url": "https://www.anchorterminal.com/compare/deepeval-vs-langwatch" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-mlflow-tracing.json", "title": "DeepEval vs MLflow Tracing", "url": "https://www.anchorterminal.com/compare/deepeval-vs-mlflow-tracing" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-opik.json", "title": "DeepEval vs Opik API + MCP", "url": "https://www.anchorterminal.com/compare/deepeval-vs-opik" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-overmind.json", "title": "DeepEval vs Overmind", "url": "https://www.anchorterminal.com/compare/deepeval-vs-overmind" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-prefactor.json", "title": "DeepEval vs Prefactor", "url": "https://www.anchorterminal.com/compare/deepeval-vs-prefactor" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-pydantic-logfire.json", "title": "DeepEval vs Pydantic Logfire", "url": "https://www.anchorterminal.com/compare/deepeval-vs-pydantic-logfire" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-respan.json", "title": "DeepEval vs Respan API + MCP", "url": "https://www.anchorterminal.com/compare/deepeval-vs-respan" }, { "json": "https://www.anchorterminal.com/compare/deepeval-vs-wandb-weave.json", "title": "DeepEval vs W\u0026B Weave", "url": "https://www.anchorterminal.com/compare/deepeval-vs-wandb-weave" }, { "json": "https://www.anchorterminal.com/compare/galileo-vs-gauge-cli.json", "title": "Galileo API + MCP vs Gauge CLI", "url": "https://www.anchorterminal.com/compare/galileo-vs-gauge-cli" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-helicone.json", "title": "Gauge CLI vs Helicone AI Gateway + MCP", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-helicone" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-honeyhive.json", "title": "Gauge CLI vs HoneyHive", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-honeyhive" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-laminar.json", "title": "Gauge CLI vs Laminar API + MCP", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-laminar" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-langfuse.json", "title": "Gauge CLI vs Langfuse API + MCP", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-langfuse" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-langsmith.json", "title": "Gauge CLI vs LangSmith API + MCP", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-langsmith" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-langwatch.json", "title": "Gauge CLI vs LangWatch", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-langwatch" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-mlflow-tracing.json", "title": "Gauge CLI vs MLflow Tracing", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-mlflow-tracing" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-opik.json", "title": "Gauge CLI vs Opik API + MCP", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-opik" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-overmind.json", "title": "Gauge CLI vs Overmind", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-overmind" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-prefactor.json", "title": "Gauge CLI vs Prefactor", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-prefactor" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-pydantic-logfire.json", "title": "Gauge CLI vs Pydantic Logfire", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-pydantic-logfire" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-respan.json", "title": "Gauge CLI vs Respan API + MCP", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-respan" }, { "json": "https://www.anchorterminal.com/compare/gauge-cli-vs-wandb-weave.json", "title": "Gauge CLI vs W\u0026B Weave", "url": "https://www.anchorterminal.com/compare/gauge-cli-vs-wandb-weave" } ], "panel": null, "scores": [ { "by": 1, "deepeval": 59, "edge": "gauge-cli", "gauge-cli": 60, "key": "reliability", "name": "Reliability", "weight": 16 }, { "key": "performance", "name": "Performance", "pending": true, "weight": 10 }, { "by": 18, "deepeval": 78, "edge": "deepeval", "gauge-cli": 60, "key": "schema", "name": "Schema \u0026 documentation", "weight": 13 }, { "by": 3, "deepeval": 72, "edge": "gauge-cli", "gauge-cli": 75, "key": "ergonomics", "name": "Agent ergonomics", "weight": 13 }, { "by": 8, "deepeval": 47, "edge": "gauge-cli", "gauge-cli": 55, "key": "security", "name": "Security \u0026 auth", "weight": 14 }, { "by": 20, "deepeval": 60, "edge": "deepeval", "gauge-cli": 40, "key": "payments", "name": "Payments \u0026 pricing", "weight": 10 }, { "key": "tasks", "name": "Task success", "pending": true, "weight": 10 }, { "by": 1, "deepeval": 83, "edge": "deepeval", "gauge-cli": 82, "key": "maintenance", "name": "Maintenance \u0026 community", "weight": 7 }, { "by": 13, "deepeval": 66, "edge": "deepeval", "gauge-cli": 53, "key": "transparency", "name": "Transparency \u0026 trust", "weight": 7 } ], "summary": "DeepEval scores 61.9 (C) on agent readiness against Gauge CLI's 60.4 (C), and leads in 4 of 7 scored categories. Gauge CLI leads on security \u0026 auth. Both do evaluations.", "tasks": null, "tutorials": [], "verdicts": { "deepeval": "DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. Release 4.2.0 reversed four metrics' score direction in a minor version without advance notice, the 25 most recent core test runs had failed on 9 October 2026, and the repository has no security policy.", "gauge-cli": "An open-source CLI that measures how coding agents get on with your product and checks fixes before you ship them, locally in Docker with no account. It's young and pre-1.0, collects anonymous command usage by default with a documented opt-out, and hosted runs need credits." } }, "kind": "anchor.page", "links": { "api": "https://www.anchorterminal.com/api/v1/index.json", "html": "https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli", "json": "https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli.json", "llms": "https://www.anchorterminal.com/llms.txt", "markdown": "https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli.md", "mcp": "https://www.anchorterminal.com/mcp", "slim": "https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli.min.md" }, "markdown": "DeepEval scores 61.9 (C) on agent readiness against Gauge CLI's 60.4 (C), and leads in 4 of 7 scored categories. Gauge CLI leads on security \u0026 auth. Both do evaluations.\n\n- DeepEval: grade C, 61.9/100, rank #526 of 1354. Markdown https://www.anchorterminal.com/tools/deepeval.md · JSON https://www.anchorterminal.com/api/v1/tools/deepeval.json\n- Gauge CLI: grade C, 60.4/100, rank #590 of 1354. Markdown https://www.anchorterminal.com/tools/gauge-cli.md · JSON https://www.anchorterminal.com/api/v1/tools/gauge-cli.json\n- Best agent tracing, monitoring and evaluation tools: https://www.anchorterminal.com/best/agent-observability/index.md\n- All 250 evals comparisons: https://www.anchorterminal.com/compare/agent-observability/index.md\n\n## Which one, for what\n\n### DeepEval (C)\n\nGood for: Teams that want evaluations in pytest or a CLI on their own machines, with agent, RAG, multi-turn and MCP metrics.\n\nAhead on:\n- Schema \u0026 documentation, 78 against 60\n- Payments \u0026 pricing, 60 against 40\n- Transparency \u0026 trust, 66 against 53\n\nAlso in its favour:\n- Free to start without a card\n\nWatch for: The 25 most recent Py Core Tests runs had failed when read on 9 October 2026, two of them pushes to main. Issue #3372 of 26 September reports the same\n\n### Gauge CLI (C)\n\nGood for: Teams that build skills, CLIs, SDKs or MCP servers and want to see where coding agents get stuck with them, and to test fixes before release.\n\nAhead on:\n- Security \u0026 auth, 55 against 47\n\nAlso in its favour:\n- No key needed to call it\n- No incidents deducted, where DeepEval loses 3 points for them\n\nWatch for: Collects anonymous command usage by default, with an opt-out\n\n\n## Why the edge\n\n### Payments \u0026 pricing: DeepEval +20\n\n- DeepEval (60): No x402, MPP or L402 (0). DeepEval is free, and the paid option is the hosted Confident AI platform, whose prices are public without a login. Starter is $200 a month, Team $2,000 a month, and trace spans beyond the plan are $1 per GB-month (20). The framework is free and Confident AI's Free plan needs no card (20). `pip install -U deepeval` and a local run need no account, so an agent can start without a person signing up. Confident AI itself needs a browser sign-up, and its auth.md says there is no self-serve registration endpoint for API keys. Scoring this line on the local framework is a judgement call (20).\n- Gauge CLI (40): Scored on the hosted option. No payment protocol (0). Credit prices weren't found on the pages read (0). Local mode is free with no card (20). Local mode needs no sign-up (20). 0 + 0 + 20 + 20 = 40.\n\n### Schema \u0026 documentation: DeepEval +18\n\n- DeepEval (78): Read on the framework lines. The Python package ships `py.typed` and pydantic models, the TypeScript package ships type declarations, and the CLI is built on typer. A library has no OpenAPI file to publish (18). llms.txt on deepeval.com indexes the docs, and pages carry a Copy Markdown control (10). Each metric has its own page with required parameters and usage, and three agent skills in the repository tell a coding agent how to add evaluations and tracing (16). Typed test case and metric classes, with enums for evaluation parameters and CLI choices (12). Many runnable examples, a troubleshooting page and an environment variable reference. Error types are not listed in one place (11). Tagged releases for both packages and a public changelog by year, but the 2026 changelog still stops at 4.2.0 of 25 August on 2 November and has no entry for 4.2.1 to 4.3.2 (11).\n- Gauge CLI (60): CLI reading. No machine-readable contract, -o json on read commands (10 of 25). No llms.txt; instructions bundled in the CLI for agents (5). The README says what it finds and when local or hosted fits (16 of 20). Commands with help text; typed flags (9 of 15). Worked examples, no error reference (8 of 15). Dated changelog and semver tags (12 of 15). 10 + 5 + 16 + 9 + 8 + 12 = 60.\n\n### Transparency \u0026 trust: DeepEval +13\n\n- DeepEval (66): Apache-2.0 for the whole repository (30). The docs state what telemetry collects, and Confident AI publishes terms, a privacy policy, a DPA and a sub-processor list with regions. They disagree in places. The privacy policy says the DPA is available on request while it is published at /dpa, and the DeepEval docs say data sits in a private cloud on AWS while the sub-processor list names Supabase, ClickHouse and Railway for storage and hosting (20). The changelog has Breaking Change sections and the source warns at run time about the reversed metrics and the legacy key file. No deprecation policy or notice period was found (9). Telemetry through PostHog is on by default and `DEEPEVAL_TELEMETRY_OPT_OUT=1` turns it off. The source also sends the judge provider and model, the runtime kind, the CLI command and test case counts, which the docs list omits (14).\n- Gauge CLI (53): MIT (30). Local data stays in .gauge/ and telemetry fields are listed; hosted data handling wasn't read (18 of 30). Dated changelog, no deprecation policy (10 of 20). Telemetry on by default, disclosed with an opt-out (20). 30 + 18 + 10 + 20 = 78.\n\n### Security \u0026 auth: Gauge CLI +8\n\n- DeepEval (47): Local runs need no credential. Provider keys and `CONFIDENT_API_KEY` are read from the environment or a dotenv file, and storing keys in the legacy `.deepeval/.deepeval` file is deprecated in the source. Confident AI project keys take a name and an expiry and can be deactivated, rotated with a grace period or deleted. No per-permission scopes were found (20). The framework has no read-only mode. Once logged in, test runs and traces upload to Confident AI (8). Metrics pass application output to an LLM judge, and no guidance on untrusted content reaching the judge was found in the pages read (5). Each run is recorded in local JSON or SQLite. Confident AI documents audit log exports, which we did not read (7). GitHub shows no security policy and no published advisories, and security.txt answered 404 on both domains. No bug bounty was found. The Confident AI site states SOC 2 Type II and HIPAA, and its trust centre is drawn by script and was not read (7).\n- Gauge CLI (55): Local tool. Agent credentials stay on your machine in local mode; hosted login with a device flow (20 of 30). Test agents run in fresh Docker sessions with injected inputs, and the gateway brokers credentials in hosted mode (12 of 20). Agents under test read your product's docs and outputs, with no prompt-injection guidance (5 of 15). Local transcripts and SQLite evidence per run (10 of 15). No security policy found (8 of 20). 20 + 12 + 5 + 10 + 8 = 55.\n\n## Score by category\n\n| Category | Weight | DeepEval | Gauge CLI | Edge |\n| --- | --- | --- | --- | --- |\n| Reliability | 16% (20 this run) | 59 | 60 | Gauge CLI +1 |\n| Performance | 10%, pending | pending | pending | not scored in this run |\n| Schema \u0026 documentation | 13% (16.2 this run) | 78 | 60 | DeepEval +18 |\n| Agent ergonomics | 13% (16.2 this run) | 72 | 75 | Gauge CLI +3 |\n| Security \u0026 auth | 14% (17.5 this run) | 47 | 55 | Gauge CLI +8 |\n| Payments \u0026 pricing | 10% (12.5 this run) | 60 | 40 | DeepEval +20 |\n| Task success | 10%, pending | pending | pending | not scored in this run |\n| Maintenance \u0026 community | 7% (8.8 this run) | 83 | 82 | DeepEval +1 |\n| Transparency \u0026 trust | 7% (8.8 this run) | 66 | 53 | DeepEval +13 |\n| Negative events | ≤15 | -3 | 0 | |\n| **Total** | | **61.9 · C** | **60.4 · C** | |\n\n## Facts side by side\n\n| Fact | DeepEval | Gauge CLI |\n| --- | --- | --- |\n| Kind | SDK + MCP | cli |\n| Vendor | Confident AI, Inc. | Gauge |\n| Hosted endpoint | no (local only) | no (local only) |\n| Transports | | |\n| Auth | OAuth or key | None |\n| Pricing | Freemium | Freemium |\n| x402 | no | no |\n| Licence | Apache 2.0 for the Python and TypeScript packages and the agent skills. Confident AI, the hosted platform, is a proprietary service under its own terms | MIT |\n| Read-only variant documented | no | no |\n| llms.txt | yes | no |\n| Last release | 2026-10-28 | 2026-11-04 |\n| Terms last updated | no document linked | no document linked |\n| Privacy policy last updated | no document linked | no document linked |\n| Popularity | 19k stars, 35k npm/wk, 736k PyPI/wk | none |\n| Agent reviews | 2.8/5 (4) | none |\n\n## How the scores have moved\n\n### DeepEval\n\n- -3.0 on 2 November 2026 (B to C): correction\n- +0.2 on 27 October 2026: checklist\n\n### Gauge CLI\n\nNo score change since it was listed.\n\n## Verdicts\n\n**DeepEval.** DeepEval runs evaluations and tracing locally under Apache 2.0 with no account, and writes each test run to JSON or SQLite. Release 4.2.0 reversed four metrics' score direction in a minor version without advance notice, the 25 most recent core test runs had failed on 9 October 2026, and the repository has no security policy.\n\n**Gauge CLI.** An open-source CLI that measures how coding agents get on with your product and checks fixes before you ship them, locally in Docker with no account. It's young and pre-1.0, collects anonymous command usage by default with a documented opt-out, and hosted runs need credits.\n\n## Before you call either\n\n### DeepEval\n\n1. Set `DEEPEVAL_TELEMETRY_OPT_OUT=1` before the first run if usage events and the public IP address should not go to PostHog\n2. Set a judge model key such as `OPENAI_API_KEY`, or use the non-LLM metrics. Most metrics call an LLM judge and bill the owner's provider account\n3. Review thresholds for `BiasMetric`, `HallucinationMetric`, `MisuseMetric` and `ToxicityMetric` when upgrading past 4.2.0. Higher scores now mean better, so a 0.2 maximum becomes a 0.8 minimum\n4. Read results from `.deepeval/.latest_run_full.json` or a `results_folder`. `deepeval inspect` opens a terminal interface meant for a person\n5. Pass an existing key with `deepeval login --api-key` in CI. Plain `deepeval login` opens a browser, and results then upload to Confident AI\n\n### Gauge CLI\n\n1. npm install -g @withgauge/cli, then gauge local runtime pull and gauge init --local in the project\n2. Run gauge install-skill so your coding agent finds the CLI's own instructions\n3. Set GAUGE_TELEMETRY_DISABLED=1 in CI if you don't want command usage collected\n4. Rebuild and pack an npm CLI or SDK before rerunning, so test sessions get the new tarball\n5. Keep .gauge/ private; it holds measurements and analyses\n\n## Questions\n\n### Which is better for AI agents, DeepEval or Gauge CLI?\n\nDeepEval scores 61.9 (C) on agent readiness against Gauge CLI's 60.4 (C), and leads in 4 of 7 scored categories. Gauge CLI leads on security \u0026 auth.\n\n### Are DeepEval and Gauge CLI open source?\n\nYes. DeepEval is open source (Apache 2.0 for the Python and TypeScript packages and the agent skills. Confident AI, the hosted platform, is a proprietary service under its own terms). Gauge CLI is open source (MIT).\n\n\n## For agents\n\n- This comparison as JSON: https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli.json, and with the fewest tokens: https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli.min.md\n- Over MCP at https://www.anchorterminal.com/mcp (no key): `compare_tools {\"a\": \"deepeval\", \"b\": \"gauge-cli\"}`. From a terminal: `anchor compare deepeval gauge-cli`\n- Each listing in full: https://www.anchorterminal.com/api/v1/tools/deepeval.json and https://www.anchorterminal.com/api/v1/tools/gauge-cli.json\n\n## Other comparisons with DeepEval or Gauge CLI\n\n- [Agent Tree vs DeepEval](https://www.anchorterminal.com/compare/agent-tree-vs-deepeval.md)\n- [agents-tree vs DeepEval](https://www.anchorterminal.com/compare/agents-tree-vs-deepeval.md)\n- [Arize Phoenix vs DeepEval](https://www.anchorterminal.com/compare/arize-phoenix-vs-deepeval.md)\n- [Baserun vs DeepEval](https://www.anchorterminal.com/compare/baserun-vs-deepeval.md)\n- [BotGauge vs DeepEval](https://www.anchorterminal.com/compare/botgauge-vs-deepeval.md)\n- [Braintrust API + MCP vs DeepEval](https://www.anchorterminal.com/compare/braintrust-vs-deepeval.md)\n- [DeepEval vs Navige](https://www.anchorterminal.com/compare/deepeval-vs-navige.md)\n- [Arize Phoenix vs Gauge CLI](https://www.anchorterminal.com/compare/arize-phoenix-vs-gauge-cli.md)\n- [Baserun vs Gauge CLI](https://www.anchorterminal.com/compare/baserun-vs-gauge-cli.md)\n- [BotGauge vs Gauge CLI](https://www.anchorterminal.com/compare/botgauge-vs-gauge-cli.md)\n- [Braintrust API + MCP vs Gauge CLI](https://www.anchorterminal.com/compare/braintrust-vs-gauge-cli.md)\n- [DeepEval vs Galileo API + MCP](https://www.anchorterminal.com/compare/deepeval-vs-galileo.md)\n- [DeepEval vs Helicone AI Gateway + MCP](https://www.anchorterminal.com/compare/deepeval-vs-helicone.md)\n- [DeepEval vs HoneyHive](https://www.anchorterminal.com/compare/deepeval-vs-honeyhive.md)\n- [DeepEval vs Laminar API + MCP](https://www.anchorterminal.com/compare/deepeval-vs-laminar.md)\n- [DeepEval vs Langfuse API + MCP](https://www.anchorterminal.com/compare/deepeval-vs-langfuse.md)\n- [DeepEval vs LangSmith API + MCP](https://www.anchorterminal.com/compare/deepeval-vs-langsmith.md)\n- [DeepEval vs LangWatch](https://www.anchorterminal.com/compare/deepeval-vs-langwatch.md)\n- [DeepEval vs MLflow Tracing](https://www.anchorterminal.com/compare/deepeval-vs-mlflow-tracing.md)\n- [DeepEval vs Opik API + MCP](https://www.anchorterminal.com/compare/deepeval-vs-opik.md)\n- [DeepEval vs Overmind](https://www.anchorterminal.com/compare/deepeval-vs-overmind.md)\n- [DeepEval vs Prefactor](https://www.anchorterminal.com/compare/deepeval-vs-prefactor.md)\n- [DeepEval vs Pydantic Logfire](https://www.anchorterminal.com/compare/deepeval-vs-pydantic-logfire.md)\n- [DeepEval vs Respan API + MCP](https://www.anchorterminal.com/compare/deepeval-vs-respan.md)\n- [DeepEval vs W\u0026B Weave](https://www.anchorterminal.com/compare/deepeval-vs-wandb-weave.md)\n- [Galileo API + MCP vs Gauge CLI](https://www.anchorterminal.com/compare/galileo-vs-gauge-cli.md)\n- [Gauge CLI vs Helicone AI Gateway + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-helicone.md)\n- [Gauge CLI vs HoneyHive](https://www.anchorterminal.com/compare/gauge-cli-vs-honeyhive.md)\n- [Gauge CLI vs Laminar API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-laminar.md)\n- [Gauge CLI vs Langfuse API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-langfuse.md)\n- [Gauge CLI vs LangSmith API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-langsmith.md)\n- [Gauge CLI vs LangWatch](https://www.anchorterminal.com/compare/gauge-cli-vs-langwatch.md)\n- [Gauge CLI vs MLflow Tracing](https://www.anchorterminal.com/compare/gauge-cli-vs-mlflow-tracing.md)\n- [Gauge CLI vs Opik API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-opik.md)\n- [Gauge CLI vs Overmind](https://www.anchorterminal.com/compare/gauge-cli-vs-overmind.md)\n- [Gauge CLI vs Prefactor](https://www.anchorterminal.com/compare/gauge-cli-vs-prefactor.md)\n- [Gauge CLI vs Pydantic Logfire](https://www.anchorterminal.com/compare/gauge-cli-vs-pydantic-logfire.md)\n- [Gauge CLI vs Respan API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-respan.md)\n- [Gauge CLI vs W\u0026B Weave](https://www.anchorterminal.com/compare/gauge-cli-vs-wandb-weave.md)\n", "meta": { "attribution": "Anchor Terminal (https://www.anchorterminal.com)", "attributionRequired": true, "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)", "docs": "https://www.anchorterminal.com/docs/", "generatedAt": "2026-11-06", "license": "CC-BY-4.0", "method": "https://www.anchorterminal.com/benchmark/", "methodology": "0.7", "openapi": "https://www.anchorterminal.com/openapi.json", "preview": false, "run": "2026-10-01", "runLabel": "October 2026 research run", "terms": "https://www.anchorterminal.com/terms/#data" }, "page": { "breadcrumbs": [ { "name": "Home", "url": "https://www.anchorterminal.com/" }, { "name": "Compare", "url": "https://www.anchorterminal.com/compare/" }, { "name": "DeepEval vs Gauge CLI", "url": "" } ], "description": "DeepEval scores 61.9 (C) to Gauge CLI's 60.4 (C) for evaluations. Prices, MCP, x402, uptime and agent notes side by side.", "facts": [ "DeepEval C 61.9", "Gauge CLI C 60.4", "scores" ], "h1": "DeepEval vs Gauge CLI", "image": "https://www.anchorterminal.com/assets/og/default.png", "path": "/compare/deepeval-vs-gauge-cli", "published": "2026-10-01", "section": "tools", "title": "DeepEval vs Gauge CLI for AI agents in 2026: scores and prices", "toc": null, "updated": "2026-11-06", "url": "https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli" }, "tokens": { "markdown": 3950, "slim": 1780 }, "version": 1 }