BotGauge vs Braintrust API + MCP

BotGauge 18

No status page. status.botgauge.com doesn't resolve and neither the site nor the docs link one (0). With no history to read, 5. No request rate limits for ingest or the eval API. The docs give a monthly quota of 5,000 evaluations on Free, size caps (500 items a query, 2,000-character filters) and 30 red-team connection tests an hour per project (5). No 429 or Retry-After guidance. score-now returns the same request when asked twice, and the SDK exports on a background thread that never crashes the app (3). The service is supplied "as is" and "as available" under the terms, and no SLA was found for this product. The 99.9 per cent SLA on the /ai-agents page belongs to the QA service (0). The service isn't labelled beta, but the SDK is 0.x, marked "4 - Beta" on PyPI, and the docs say API shapes may change before 1.0 (5). 0+5+5+3+0+5 = 18.

Braintrust API + MCP 61

Statuspage at status.braintrust.dev with component history and 25 incidents since February 2026 (20). The last 90 days hold several majors. A critical run of 504s on the EU data plane on 16 July (about 30 minutes), gateway 5xx incidents on 3 August and 27 to 28 August, control plane 5xxs on 4 September, a 40-minute data-loading failure in the US on 9 September and 78 minutes of elevated errors across the US data plane API on 30 September. Most were short and the gateway is a side product, so 5 rather than 0. The only published limit is about 20 BTQL queries a minute on Starter and Pro. Ingestion and REST limits have no numbers (8). The OpenAPI spec documents 429 with a Retry-After header on 154 operations and recommends exponential backoff, the SDKs retry 408, 429 and 5xx, and events written with the same id overwrite rather than duplicate (15). Enterprise lists SLAs and guaranteed response times with no published terms (3). API and MCP are GA (10).

BotGauge 53

No OpenAPI file for the eval or red-team endpoints was found in the docs or llms-full.txt. Ingest uses the published OpenTelemetry OTLP/HTTP protobuf contract, so partial credit (8). llms.txt, llms-full.txt and Markdown for every page on Mintlify (10). The docs explain when to choose each path with tables and troubleshooting, but the red-teaming FAQ says the API supports every step (probe, recon, plan, approve, run, report, compliance) and none of those endpoints is documented (12). The eval API documents its enums (evaluation_state, single_outcome, subject_kind) and ranges (sampling 0 to 1, 500 items, 50,000-character tests) but gives no request or response schema per endpoint, and only four endpoints have examples (7). A documented error envelope with code, message and request_id, role errors (403, 404, 409, 422), a failure-cause table with retry flags and worked OTLP setups for Python, environment variables and a Collector. No list of error codes (11). /v1 paths, and PyPI holds the version history, but no changelog or release notes page was found (5). 8+10+12+7+11+5 = 53.

Braintrust API + MCP 85

OpenAPI 3.0.3 with 75 paths and 234 operations at /docs/openapi.json, pinned and refreshed weekly in the Python SDK repo (25). llms.txt with about 500 entries, every page served as Markdown (10). The MCP docs give a one-line purpose for each of the 42 tools and mark the test_* tools as dry runs, but say little about when not to use a tool. The MCP server is closed source, so we read the docs rather than the definitions (13). Typed OpenAPI schemas with pagination parameters and enums, though sql_query takes free-form SQL and event payloads are free-form by nature (11). 400, 401, 403, 429 and 500 responses are declared on most operations, but only 4 of 234 operations carry an inline example. The cookbook fills some of the gap (11). Versioned /v1 API, a monthly product changelog, a data-plane changelog and per-SDK changelogs (15).

BotGauge 10

No x402 or other machine payment (0). Paid plans for the agent product have no published prices, and botgauge.com/pricing prices the separate QA service (0). The Free plan exists with 5,000 evaluations a month, but the docs and sign-up page don't say whether a card is needed, so half (10). A person signs up in a browser at cloud.botgauge.com and creates keys in the console (0). 0+0+10+0 = 10.

Braintrust API + MCP 40

No x402 or other machine payment (0). Usage prices published without login, $4 a GB and $2.50 per 1,000 scores on Starter, $3 and $1.50 on Pro (20). Starter is free with no card (20). A person signs up in a browser to create a key. The MCP OAuth flow also needs a browser sign-in (0).

BotGauge 67

botgauge, botgauge-tracing and botgauge-cli 0.1.3 on 2026-10-31 (30). Four releases of each since 20 October 2026, 0.1.0, 0.1.1, 0.1.2 and 0.1.3 (20). Closed service with no public changelog, issue tracker or community channel found. Support by email at support@botgauge.com (4). One official SDK, Python, current (10). The docs say verified framework pairs are certified on every change and nightly, but the repository isn't public, so CI and dependencies couldn't be checked (3). 30+20+4+10+3 = 67. The product is 12 days old by its first package release.

Braintrust API + MCP 85

TypeScript SDK braintrust 3.36.0 and Python SDK 0.44.0, both on 2026-10-01 (30). 15 TypeScript and 19 Python releases since 16 July (20). The product changelog lists dozens of September entries and both SDK repositories take outside contributions credited in the changelog. We didn't load issue reply times (12). Official SDKs in six languages, current (15). Public CI (checks.yaml), integration tests, Dependabot and a weekly dependency job. We didn't confirm the pass state (8).

BotGauge 55

API responses can be narrowed by subject_ids and subject_kind, a results query takes up to 500 items, and judges read at most 20,000 characters, but no field selection or summaries are documented (12). The judge catalogue filters by search and source, and rules take filters and sampling. No pagination is documented on any list endpoint (10). Errors carry a code, a message and a request id. Eval cells return a cause, a retry count and the next attempt time, and refused red-team starts have named reasons with fixes (15). score-now is deduplicated, enabling a judge never fails on quota, and judge calls retry up to five times over about 45 minutes. No idempotency keys on other writes (8). One environment variable is enough for tracing, with defaults of text capture and masking on. Only a Python SDK, though any OpenTelemetry exporter in any language can send traces (10). 12+10+15+8+10 = 55.

Braintrust API + MCP 66

42 MCP tools load at once, with no toolsets or server-side allowlist found (5). REST lists take limit, starting_after and ending_before, and sql_query truncates field values to 1,024 characters by default and swaps results over 1 MB for a signed overflow_url (20). Error codes are declared per operation, the MCP server returns 413 above 100 MiB, and 429 carries Retry-After. Error bodies are typed as plain text (14). Event inserts upsert on id and ACL batch updates are documented as idempotent. Several MCP tools have a no-write test_* twin. We couldn't see whether the hosted tools carry readOnlyHint or destructiveHint (12). Few required parameters, and SDKs in TypeScript, Python, Go, Java, Ruby and C# (15).