Category · Developer & infrastructure
Agent tracing, monitoring and evaluation tools
Tools that trace agent runs, tool calls and model calls, and run evaluations on them. Compared on what appears in a trace, how fast a failure can be found, whether an evaluation can be reproduced, setup time and cost.
Best agent tracing, monitoring and evaluation tools, the shortlist250 head-to-head comparisons
Capability keys obs.traces · obs.evals · obs.redteam · obs.prompts · obs.gateway · obs.datasets · All tools
letme.dev/obs.traces picks the top-graded tool in this list and says how to call it direct (how letme works).
The same listing from the live API. Graded results come first, then the official MCP registry when no graded-only filter is set.
https://www.anchorterminal.com/api/v1/search
Filters
| Compare | # | Tool | Category | Grade | Score | Agent rating | p95 | Context | Price / x402 | Auth | Where | Details |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 104 | Arize PhoenixArize AI · HTTP API | Evals | BB | 75.8 | 3.8 (8) | n/a | n/a | Free · OSS | OAuth or key | Local | ||
|
Self-hosted tracing, evaluation, datasets, experiments and prompt management built on OpenTelemetry and OpenInference. Top strength Free and self-hosted with no feature gating, from Top weakness Auth stays off on a loopback bind, and the default admin password is |
||||||||||||
| 147 | Langfuse API + MCPLangfuse (ClickHouse) · HTTP API | Evals | BB | 74.3 | 4.0 (2) | n/a | n/a | $29 / mo | API key | Hosted | ||
|
Open-source tracing, evaluation, prompt management and datasets for LLM apps and agents, built on OpenTelemetry. Top strength MIT core with no usage limits when self-hosted, and self-hosted telemetry documented with an off switch Top weakness 98 MCP tools load by default, writes included, and the MCP docs still leave read-only to a client allowlist |
||||||||||||
| 152 | LangSmith API + MCPLangChain · HTTP API | Evals | BB | 74.1 | 3.0 (2) | n/a | n/a | $39 / seat-mo | OAuth or key | Hosted | ||
|
LangChain's hosted tracing, evaluation, prompt hub and dataset platform, framework-agnostic with Python and TypeScript SDKs and OpenTelemetry ingest. Top strength Rate limits published per endpoint and per plan, with each kind of 429 explained Top weakness Five SDK advisories from February to June 2026, one listed as critical (arbitrary file read through |
||||||||||||
| 162 | Respan API + MCPRespan (formerly Keywords AI) · HTTP API | Evals | BB | 73.9 | 2.5 (2) | n/a | n/a | $199 / mo | OAuth or key | Hosted | ||
|
Hosted tracing, evals, prompt management and an LLM gateway to 250+ models, from the company formerly called Keywords AI. Top strength API keys with Read, Write, Gateway and Admin scopes, expiry and spend caps, bound to one project Top weakness Security fixes on 30 July, 28 August and 19 October 2026 disclosed only as one-line changelog entries |
||||||||||||
| 164 | Opik API + MCPComet · HTTP API | Evals | BB | 73.8 | 3.5 (4) | n/a | n/a | $19 / mo | OAuth or key | Hosted and local | ||
|
Opik is Comet's open-source platform for tracing, evaluating and optimising LLM apps and agents, under Apache-2.0. It runs self-hosted or on Comet's Opik Cloud, with a REST API, Python and TypeScript SDKs and a hosted MCP server. Top strength Five MCP tools carry readOnlyHint and destructiveHint, Top weakness No read-only mode or per-project scope for the MCP server, and the OAuth metadata advertises no scopes |
||||||||||||
| 390 | LangWatchReasoning Engine B.V. (LangWatch) · HTTP API | Evals | B | 68.1 | 3.0 (4) | n/a | n/a | Freemium | OAuth or key | Hosted and local | ||
|
LangWatch is an open-source platform for tracing, evaluating and testing LLM applications and agents, with prompt management, datasets and an AI gateway. It runs hosted or self-hosted, with a REST API, SDKs, a CLI and an MCP server. Top strength API keys take read or write per permission category, a project, team or organisation scope and an expiry. Ingestion keys can only write traces Top weakness The MCP server registers 110 tools, deletes and key creation among them, with no toolsets and no |
||||||||||||
| 402 | agents-treeNot for agents Karel Blavka · Agent framework | Evals | B | 68 | none | n/a | n/a | Free · OSS | None | Library | ||
|
agents-tree is an MIT-licensed Python CLI, with an optional herdr plugin, that shows Claude Code, Codex, Grok Build and Antigravity CLI sessions and their sub-agents as a live tree, read from the transcripts on disk. Top strength Read-only by design. It never writes to a session, opens SQLite files read-only and has no network code Top weakness One maintainer, first released on 6 October 2026, with no company, site or security policy |
||||||||||||
| 421 | Pydantic LogfirePydantic · HTTP API | Evals | B | 67.6 | 3.3 (4) | n/a | n/a | $49 / mo | OAuth or key | Hosted | ||
|
Pydantic Logfire is a hosted observability platform built on OpenTelemetry for traces, logs, metrics, LLM cost tracking, evaluations and prompt management. Agents reach it through a remote MCP server, a SQL query API, a public REST API and SDKs. Top strength The remote MCP server uses OAuth with PKCE (S256 required) and dynamic client registration, or a Bearer API key with as little as the Top weakness No public status page was found on pydantic.dev, in the docs or in the files published for agents |
||||||||||||
| 422 | Braintrust API + MCPBraintrust · HTTP API | Evals | B | 67.6 | 3.0 (2) | n/a | n/a | $249 / mo | OAuth or key | Hosted | ||
|
Hosted tracing, logging and evaluation for LLM apps and agents, with experiments, datasets, prompts, online scorers and a model gateway. Top strength OpenAPI 3.1.1 spec with 77 paths in a public Apache-2.0 repository, 429 and Top weakness The hosted |
||||||||||||
| 507 | DeepEvalConfident AI, Inc. · SDK + MCP | Evals | B | 65.7 | 2.8 (4) | n/a | n/a | $200 / mo | OAuth or key | Local | ||
|
DeepEval is an open-source Python and TypeScript framework from Confident AI for evaluating LLM applications and agents. Evaluations run locally through a pytest plugin and the `deepeval` CLI, with optional reporting to the hosted Confident AI platform. Top strength Apache-2.0 framework that runs evaluations and tracing locally with no account. Test runs are written to JSON files or a SQLite database on the owner's disk Top weakness The 25 most recent Py Core Tests runs had failed when read on 9 October 2026, two of them pushes to main. Issue #3372 of 26 September reports the same |
||||||||||||
| 510 | W&B WeaveWeights & Biases (CoreWeave) · HTTP API | Evals | B | 65.7 | 2.8 (4) | n/a | n/a | $60 / mo | API key | Hosted | ||
|
W&B Weave is CoreWeave Forge's tracing and evaluation service for LLM applications and agents. It takes traces from Python and TypeScript SDKs or an OTLP endpoint and exposes them through a REST Service API. Top strength A live OpenAPI 3.1 document at Top weakness No request rate limits for the multi-tenant Service API were found in the reviewed documentation |
||||||||||||
| 591 | PrefactorPrefactor Pty Ltd · HTTP API | Evals | B | 64 | 3.0 (4) | n/a | n/a | $49 / mo | API key | Hosted | ||
|
Prefactor is a hosted service that records AI agent runs as spans, classifies each run's data risk and stores quality evaluations. Agents and scripts reach it through an HTTP and WebSocket API, TypeScript and Python SDKs and a CLI. Top strength Public OpenAPI 3.0 document with 163 operations and an OpenRPC document with 165 WebSocket methods, both fetched without a login Top weakness The pricing page lists hold, approve or block and PII detection on every plan, while the docs say a risk threshold blocks nothing and that Prefactor does not detect sensitive data |
||||||||||||
| 643 | Gauge CLIGauge · SDK + MCP | Evals | B | 62.9 | none | n/a | n/a | Freemium | None | Local | ||
|
Gauge CLI runs real coding agents (Claude Code, Codex, Pi, OpenCode) against your skills, CLIs, SDKs, MCP servers and docs in Docker, records where they get stuck, and lets your agent test fixes before release. Top strength Runs Claude Code, Codex, Pi and OpenCode in fresh Docker sessions against working-tree or pre-release versions Top weakness Collects anonymous command usage by default, with an opt-out |
||||||||||||
| 763 | OvermindOvermind Technology Ltd · HTTP API | Evals | C | 60.3 | none | n/a | n/a | $120 / mo | OAuth or key | Hosted | ||
|
Overmind is a hosted and self-hostable platform that traces AI agents, turns traces and files into versioned datasets, runs evaluations, fine-tunes open-weight models and serves them on an OpenAI-compatible API, driven from a coding agent over MCP, CLI or REST. Top strength Platform under AGPL-3.0 and SDK and CLI under MIT, with the same code self-hostable by Docker Compose Top weakness No status page, rate limits with numbers or 429 guidance was found |
||||||||||||
| 814 | Laminar API + MCPLaminar (LMNR AI) · HTTP API | Evals | C | 58.9 | 3.5 (2) | n/a | n/a | $30 / mo | API key | Hosted | ||
|
Open-source tracing, evaluations and an agent debugger built on OpenTelemetry, with browser session recordings synced to traces for browser agents. Top strength Browser session recordings synced to traces for Browser Use, Stagehand, Playwright and Puppeteer Top weakness A cross-tenant SQL execution and export hole was fixed on 27 August 2026 in a commit, with no advisory |
||||||||||||
| 933 | Agent TreeNot for agents Mike Pollow (Digitalroots) · Agent framework | Evals | C | 56.2 | none | n/a | n/a | Free · OSS | None | Library | ||
|
Agent Tree is an MIT-licensed Claude Code plugin that shows a session's sub-agents as a live tree in a terminal side pane, with context, cost and usage limits, and lets the person stop, message or re-model running sub-agents. Top strength No account, key or telemetry; the one network request is a daily version check, disclosed in PRIVACY.md with five ways to turn it off Top weakness Version 0.5.4, ten days old, written and released by one maintainer with no company, site or register entry found |
||||||||||||
| 969 | MLflow TracingMLflow Project (LF Projects, LLC) · HTTP API | Evals | C | 55.2 | 3.0 (4) | n/a | n/a | Free · OSS | OAuth or key | Local | ||
|
Open-source tracing, evaluation and prompt management for LLM applications and agents, part of MLflow, a Linux Foundation project. Owners run the server themselves, and agents read and annotate traces through an experimental MCP server or the `mlflow traces` CLI. Top strength Apache-2.0 licence, free to self-host, with nothing to buy from the project Top weakness The MCP server is marked experimental in the docs and sets no |
||||||||||||
| 1040 | HoneyHiveHoneyHive · HTTP API | Evals | D | 53 | 2.5 (2) | n/a | n/a | Freemium | API key | Hosted | ||
|
Hosted tracing, evaluation, datasets and prompt management for agents, built on OpenTelemetry. Top strength Read-only ( Top weakness Status page showed its single component in Downtime with 0.000 per cent uptime on 4 November 2026, with no incident history |
||||||||||||
| 1235 | Galileo API + MCPGalileo (now Splunk Agent Observability, Cisco) · HTTP API | Evals | D | 46.1 | 2.5 (2) | n/a | n/a | $100 / mo | API key | Hosted | ||
|
Hosted tracing, evaluation metrics and guardrails for LLM apps and agents, with a REST API, Python and TypeScript SDKs and an 8-tool MCP server in preview. Top strength OpenAPI 3.1 spec with 184 paths and 244 operations, linked from the docs' llms.txt Top weakness No status page, no published rate limits and no SLA |
||||||||||||
| 1273 | Helicone AI Gateway + MCPHelicone · HTTP API | Evals | E | 44.6 | 2.0 (2) | n/a | n/a | $79 / mo | API key | Hosted and local | ||
|
Open-source LLM observability that logs requests through an OpenAI-compatible gateway (100+ models, fallbacks, caching, rate limits) or async logging. Top strength One base-URL change gives logging, caching, fallbacks and rate limits for 100+ models Top weakness Security fixes from 30 August to 19 October 2026, including cross-tenant access to decrypted provider keys, disclosed only in commit messages |
||||||||||||
| 1328 | NavigeNavige Ltd · HTTP API | Evals | E | 41.6 | 2.0 (4) | n/a | n/a | $9 / mo | API key | Hosted | ||
|
Navige is a hosted gateway between AI agents and the model APIs and MCP tools they call. It prices each call per business job, enforces spend caps, policies and approvals, and can cache or reroute calls to cut cost. Top strength Joining is a base-URL change and one header. The provider key is forwarded untouched, and prompt and completion text is not stored, per the docs and DPA 1.2 Top weakness The Terms (version 1.0) leave exact prices, payment terms, liability, service levels and governing law marked TBD |
||||||||||||
| 1336 | BotGaugeBotGauge AI Inc. · HTTP API | Evals | E | 41.3 | none | n/a | n/a | Freemium | API key | Hosted | ||
|
BotGauge is a hosted platform that red-teams AI agents through their HTTP endpoint, ingests OpenTelemetry traces and scores them with AI judges. A Python SDK, an eval REST API and a free plan are available. Top strength Accepts plain OTLP/HTTP from any OpenTelemetry exporter or Collector, so tracing needs no BotGauge code Top weakness First SDK release on 20 October 2026, with 0.x packages marked Beta whose API may change before 1.0 |
||||||||||||
| shut | BaserunShut down 2024-09-30 Baserun · HTTP API | Evals | F | 6.4 | 1.0 (2) | n/a | n/a | $49 / seat-mo | API key | Hosted | ||
|
Discontinued LLM testing and monitoring platform with Python and JavaScript SDKs for tracing model calls. Top strength SDKs were MIT licensed and the Python source is still public at github.com/baserun-ai/baserun-py Top weakness The service has closed; the co-founder lists Baserun as acquired and www.baserun.ai fails the TLS handshake |
||||||||||||
Nothing matches these filters. .
p95 latency and context cost come from our probes, which haven't run yet, so those columns start hidden. Grades run from AA to F, and agent-ready means BB or better. Filters, sorting and export run in your browser; the table is complete without JavaScript.
Indexed, not reviewed (5)
Listings sorted into this category from public catalogues (the official MCP registry, APIs.guru, the x402 Bazaar and OpenRouter), with facts and our own checks but no score, grade or rank. How the index works.
| Listing | Kind | What it does | Why it's here |
|---|---|---|---|
| aegis undercurrentholdings.com | MCP server | Local stdio MCP server for the AEGIS governance client. The hosted evaluation service is offline. | vendor's own |
| argosvix.com MCP server argosvix.com | MCP server | Observability MCP server: query LLM cost/errors/latency & operate alerts/evals from Claude/Cursor | vendor's own |
| Model Finder indigloo.co | MCP server | Find the cheapest AI model that passes your tests: audit repo models, build evals, routing, sizing | vendor's own |
| openfeature.dev MCP server openfeature.dev | MCP server | MCP server providing OpenFeature SDK installation guides and OFREP flag evaluation | vendor's own |
| PromptOT promptot.com | MCP server | Manage, version, and publish LLM prompts with blocks, variables, and evaluations. | vendor's own |
How we test this category
The same agent run instrumented in every tool, including a failed tool call and a retry. We check which steps appear in the trace, whether the failure can be found quickly, whether an evaluation can be reproduced, how long setup takes and what it costs.
How the ranking works
Every tool is scored 0 to 100 across 9 weighted categories, minus deductions for negative events, and given a grade from AA to F. Reliability, latency and context cost are measured by probes and by traffic through letme. Schema quality, security and payments are assessed against a published checklist. Task success comes from a fixed task suite. Payments, including x402, are one category of the nine. Read the methodology.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.