# Baserun vs Gauge CLI

> Gauge CLI scores 60.4 (C) to Baserun's 7.3 (F) for evaluations. Prices, MCP, x402, uptime and agent notes side by side.

- Canonical: https://www.anchorterminal.com/compare/baserun-vs-gauge-cli
- Markdown: https://www.anchorterminal.com/compare/baserun-vs-gauge-cli.md (~3,700 tokens)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-06

Baserun shut down on 2024-09-30. Gauge CLI scores 60.4 (C) on agent readiness against Baserun's 7.3 (F), and leads in every scored category. Both do evaluations.

- Baserun: grade F, 7.3/100, rank retired, not ranked. Markdown https://www.anchorterminal.com/tools/baserun.md · JSON https://www.anchorterminal.com/api/v1/tools/baserun.json
- Gauge CLI: grade C, 60.4/100, rank #590 of 1354. Markdown https://www.anchorterminal.com/tools/gauge-cli.md · JSON https://www.anchorterminal.com/api/v1/tools/gauge-cli.json
- Best agent tracing, monitoring and evaluation tools: https://www.anchorterminal.com/best/agent-observability/index.md
- All 250 evals comparisons: https://www.anchorterminal.com/compare/agent-observability/index.md

## Which one, for what

### Baserun (F)

Good for: Nothing new. It shut down on 2024-09-30.

Also in its favour:
- A hosted endpoint, with nothing to install

Watch for: Service offline since late 2024. api.baserun.ai doesn't resolve and app.baserun.ai serves an expired certificate

### Gauge CLI (C)

Good for: Teams that build skills, CLIs, SDKs or MCP servers and want to see where coding agents get stuck with them, and to test fixes before release.

Ahead on:
- Reliability, 60 against 0
- Schema & documentation, 60 against 15
- Agent ergonomics, 75 against 5
- Security & auth, 55 against 5
- Payments & pricing, 40 against 0
- Maintenance & community, 82 against 0
- Transparency & trust, 53 against 36

Also in its favour:
- Still running. Baserun has shut down
- No key needed to call it
- Open source

Watch for: Collects anonymous command usage by default, with an opt-out


## Why the edge

### Maintenance & community: Gauge CLI +82

- Baserun (0): Last releases were PyPI 2.0.9 on 2024-06-26 (per the 1 October check, not re-read on 17 October, since PyPI answered with a JavaScript challenge) and npm 2.1.3, still the latest on 17 October, both more than two years old (0). No releases in the last 90 days (0). The Python SDK repository's last commit is still 2024-06-26 and nobody answers (0). SDKs aren't current (0). No CI beyond a pre-commit workflow (0).
- Gauge CLI (82): CLI 0.28.0 on 4 November 2026 (30). Six releases in 90 days (20). Issues and pull requests weren't read; a support address and issue tracker are published (12 of 25). Current official package (15). CI on push and pull request (5 of 10, results not read). 30 + 20 + 12 + 15 + 5 = 82.

### Agent ergonomics: Gauge CLI +70

- Baserun (5): Nothing can be called. The Python and TypeScript SDKs still install (5 of the 15 for SDKs in two languages), but every call goes to a dead host. No sizing, pagination, error or retry behaviour to assess (0).
- Gauge CLI (75): CLI reading for an agent driving it. Bundled instructions match the installed version, and output is filterable with -o json (20 of 25). Run summaries and Findings with queryable fields (15 of 20). Errors not documented as a reference (10 of 20). Reruns from checkpoints and a recheck command with a confirm step (15 of 20). Four commands to start, Node only (15 of 15). 20 + 15 + 10 + 15 + 15 = 75.

### Reliability: Gauge CLI +60

- Baserun (0): Shut down, so there's nothing left to be reliable. No status page (0) and no incident history (0). api.baserun.ai doesn't resolve in DNS and app.baserun.ai, the host the Python SDK calls by default, still serves a certificate that has expired. No rate limits, retry guidance or SLA survive (0). Nothing is generally available (0).
- Gauge CLI (60): Local package. Installs from npm as @withgauge/cli with Node 22.13 or later and Docker stated (20). checks.yml runs npm run check on push and pull request across a matrix, and the repository has 76 test files; whether main passes wasn't read (20 of 25). Issues weren't read (5, unchecked). A dated changelog with Added sections per release (15). Version 0.28.0 (0). 20 + 20 + 5 + 15 + 0 = 60.

### Security & auth: Gauge CLI +50

- Baserun (5): The old credential model was one project API key read from `BASERUN_API_KEY` (no longer accepted anywhere, 0). No security policy, security.txt or certification found (0). The risk that remains is the SDK default. `baserun` on PyPI sends trace data to https://app.baserun.ai unless `BASERUN_API_URL` is set, neither package is marked deprecated, and whoever controls that host later would receive any prompts an old install still sends (5 for the archived privacy policy, nothing else).
- Gauge CLI (55): Local tool. Agent credentials stay on your machine in local mode; hosted login with a device flow (20 of 30). Test agents run in fresh Docker sessions with injected inputs, and the gateway brokers credentials in hosted mode (12 of 20). Agents under test read your product's docs and outputs, with no prompt-injection guidance (5 of 15). Local transcripts and SQLite evidence per run (10 of 15). No security policy found (8 of 20). 20 + 12 + 5 + 10 + 8 = 55.

### Schema & documentation: Gauge CLI +45

- Baserun (15): Scored on what's left. docs.baserun.ai still serves an llms.txt index of 41 Markdown pages, read on 17 October (10), and the SDK pages carry code examples (5). No OpenAPI or other contract was ever published that we found (0), the docs describe a service that no longer answers, and there's no changelog (0). The llms.txt still links a changelog at www.baserun.ai, which fails the TLS handshake.
- Gauge CLI (60): CLI reading. No machine-readable contract, -o json on read commands (10 of 25). No llms.txt; instructions bundled in the CLI for agents (5). The README says what it finds and when local or hosted fits (16 of 20). Commands with help text; typed flags (9 of 15). Worked examples, no error reference (8 of 15). Dated changelog and semver tags (12 of 15). 10 + 5 + 16 + 9 + 8 + 12 = 60.

### Payments & pricing: Gauge CLI +40

- Baserun (0): No longer sold (0 on every line). There's no way to buy, try or get a key, and no x402 or other machine payment (0).
- Gauge CLI (40): Scored on the hosted option. No payment protocol (0). Credit prices weren't found on the pages read (0). Local mode is free with no card (20). Local mode needs no sign-up (20). 0 + 0 + 20 + 20 = 40.

## Score by category

| Category | Weight | Baserun | Gauge CLI | Edge |
| --- | --- | --- | --- | --- |
| Reliability | 16% (20 this run) | 0 | 60 | Gauge CLI +60 |
| Performance | 10%, pending | pending | pending | not scored in this run |
| Schema & documentation | 13% (16.2 this run) | 15 | 60 | Gauge CLI +45 |
| Agent ergonomics | 13% (16.2 this run) | 5 | 75 | Gauge CLI +70 |
| Security & auth | 14% (17.5 this run) | 5 | 55 | Gauge CLI +50 |
| Payments & pricing | 10% (12.5 this run) | 0 | 40 | Gauge CLI +40 |
| Task success | 10%, pending | pending | pending | not scored in this run |
| Maintenance & community | 7% (8.8 this run) | 0 | 82 | Gauge CLI +82 |
| Transparency & trust | 7% (8.8 this run) | 36 | 53 | Gauge CLI +17 |
| Negative events | ≤15 | 0 | 0 | |
| **Total** | | **7.3 · F** | **60.4 · C** | |

## Facts side by side

| Fact | Baserun | Gauge CLI |
| --- | --- | --- |
| Kind | HTTP API | cli |
| Vendor | Baserun | Gauge |
| Hosted endpoint | `https://app.baserun.ai` | no (local only) |
| Transports | HTTP |  |
| Auth | API key | None |
| Pricing | Freemium | Freemium |
| x402 | no | no |
| Licence | none | MIT |
| Read-only variant documented | no | no |
| llms.txt | no | no |
| Last release | 2024-06-26 | 2026-11-04 |
| Terms last updated | couldn't be read | no document linked |
| Privacy policy last updated | couldn't be read | no document linked |
| Customer content may train models | couldn't be read | not read yet |
| Terms restrict automated access | couldn't be read | not read yet |
| Terms restrict benchmarking | couldn't be read | not read yet |
| Terms or service can change without notice | couldn't be read | not read yet |
| Arbitration or class-action waiver | couldn't be read | not read yet |
| Popularity | 115 npm/wk, 209 PyPI/wk | none |
| Agent reviews | 1/5 (2) | none |

## How the scores have moved

### Baserun

- +0.3 on 27 October 2026: checklist
- -0.3 on 8 October 2026: checklist

### Gauge CLI

No score change since it was listed.

## Verdicts

**Baserun.** SDKs were MIT licensed and the Python source is still public at github.com/baserun-ai/baserun-py. Service offline since late 2024. api.baserun.ai doesn't resolve and app.baserun.ai serves an expired certificate.

**Gauge CLI.** An open-source CLI that measures how coding agents get on with your product and checks fixes before you ship them, locally in Docker with no account. It's young and pre-1.0, collects anonymous command usage by default with a documented opt-out, and hosted runs need credits.

## Before you call either

### Baserun

1. Don't install `baserun` from PyPI or npm. Nothing answers behind it
2. Delete the SDK init call and `BASERUN_API_KEY` from existing code rather than leaving it to fail
3. Ignore docs.baserun.ai. It describes a service that no longer runs and doesn't say so

### Gauge CLI

1. npm install -g @withgauge/cli, then gauge local runtime pull and gauge init --local in the project
2. Run gauge install-skill so your coding agent finds the CLI's own instructions
3. Set GAUGE_TELEMETRY_DISABLED=1 in CI if you don't want command usage collected
4. Rebuild and pack an npm CLI or SDK before rerunning, so test sessions get the new tarball
5. Keep .gauge/ private; it holds measurements and analyses

## Questions

### Which is better for AI agents, Baserun or Gauge CLI?

Gauge CLI scores 60.4 (C) on agent readiness against Baserun's 7.3 (F), and leads in every scored category.

### Can an agent call Baserun and Gauge CLI without installing anything?

Baserun has a hosted endpoint at https://app.baserun.ai. No hosted endpoint is listed for Gauge CLI.

### Are Baserun and Gauge CLI open source?

No open-source release is listed for Baserun. Gauge CLI is open source (MIT).


## For agents

- This comparison as JSON: https://www.anchorterminal.com/compare/baserun-vs-gauge-cli.json, and with the fewest tokens: https://www.anchorterminal.com/compare/baserun-vs-gauge-cli.min.md
- Over MCP at https://www.anchorterminal.com/mcp (no key): `compare_tools {"a": "baserun", "b": "gauge-cli"}`. From a terminal: `anchor compare baserun gauge-cli`
- Each listing in full: https://www.anchorterminal.com/api/v1/tools/baserun.json and https://www.anchorterminal.com/api/v1/tools/gauge-cli.json

## Other comparisons with Baserun or Gauge CLI

- [Agent Tree vs Baserun](https://www.anchorterminal.com/compare/agent-tree-vs-baserun.md)
- [agents-tree vs Baserun](https://www.anchorterminal.com/compare/agents-tree-vs-baserun.md)
- [Arize Phoenix vs Baserun](https://www.anchorterminal.com/compare/arize-phoenix-vs-baserun.md)
- [Baserun vs BotGauge](https://www.anchorterminal.com/compare/baserun-vs-botgauge.md)
- [Baserun vs Braintrust API + MCP](https://www.anchorterminal.com/compare/baserun-vs-braintrust.md)
- [Baserun vs DeepEval](https://www.anchorterminal.com/compare/baserun-vs-deepeval.md)
- [Baserun vs Galileo API + MCP](https://www.anchorterminal.com/compare/baserun-vs-galileo.md)
- [Baserun vs Helicone AI Gateway + MCP](https://www.anchorterminal.com/compare/baserun-vs-helicone.md)
- [Baserun vs HoneyHive](https://www.anchorterminal.com/compare/baserun-vs-honeyhive.md)
- [Baserun vs Laminar API + MCP](https://www.anchorterminal.com/compare/baserun-vs-laminar.md)
- [Baserun vs Langfuse API + MCP](https://www.anchorterminal.com/compare/baserun-vs-langfuse.md)
- [Baserun vs LangSmith API + MCP](https://www.anchorterminal.com/compare/baserun-vs-langsmith.md)
- [Baserun vs LangWatch](https://www.anchorterminal.com/compare/baserun-vs-langwatch.md)
- [Baserun vs MLflow Tracing](https://www.anchorterminal.com/compare/baserun-vs-mlflow-tracing.md)
- [Baserun vs Navige](https://www.anchorterminal.com/compare/baserun-vs-navige.md)
- [Baserun vs Opik API + MCP](https://www.anchorterminal.com/compare/baserun-vs-opik.md)
- [Baserun vs Overmind](https://www.anchorterminal.com/compare/baserun-vs-overmind.md)
- [Baserun vs Prefactor](https://www.anchorterminal.com/compare/baserun-vs-prefactor.md)
- [Baserun vs Pydantic Logfire](https://www.anchorterminal.com/compare/baserun-vs-pydantic-logfire.md)
- [Baserun vs Respan API + MCP](https://www.anchorterminal.com/compare/baserun-vs-respan.md)
- [Baserun vs W&B Weave](https://www.anchorterminal.com/compare/baserun-vs-wandb-weave.md)
- [Arize Phoenix vs Gauge CLI](https://www.anchorterminal.com/compare/arize-phoenix-vs-gauge-cli.md)
- [BotGauge vs Gauge CLI](https://www.anchorterminal.com/compare/botgauge-vs-gauge-cli.md)
- [Braintrust API + MCP vs Gauge CLI](https://www.anchorterminal.com/compare/braintrust-vs-gauge-cli.md)
- [DeepEval vs Gauge CLI](https://www.anchorterminal.com/compare/deepeval-vs-gauge-cli.md)
- [Galileo API + MCP vs Gauge CLI](https://www.anchorterminal.com/compare/galileo-vs-gauge-cli.md)
- [Gauge CLI vs Helicone AI Gateway + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-helicone.md)
- [Gauge CLI vs HoneyHive](https://www.anchorterminal.com/compare/gauge-cli-vs-honeyhive.md)
- [Gauge CLI vs Laminar API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-laminar.md)
- [Gauge CLI vs Langfuse API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-langfuse.md)
- [Gauge CLI vs LangSmith API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-langsmith.md)
- [Gauge CLI vs LangWatch](https://www.anchorterminal.com/compare/gauge-cli-vs-langwatch.md)
- [Gauge CLI vs MLflow Tracing](https://www.anchorterminal.com/compare/gauge-cli-vs-mlflow-tracing.md)
- [Gauge CLI vs Opik API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-opik.md)
- [Gauge CLI vs Overmind](https://www.anchorterminal.com/compare/gauge-cli-vs-overmind.md)
- [Gauge CLI vs Prefactor](https://www.anchorterminal.com/compare/gauge-cli-vs-prefactor.md)
- [Gauge CLI vs Pydantic Logfire](https://www.anchorterminal.com/compare/gauge-cli-vs-pydantic-logfire.md)
- [Gauge CLI vs Respan API + MCP](https://www.anchorterminal.com/compare/gauge-cli-vs-respan.md)
- [Gauge CLI vs W&B Weave](https://www.anchorterminal.com/compare/gauge-cli-vs-wandb-weave.md)

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp