Category · Developer & infrastructure

Agent tracing, monitoring and evaluation tools

Tools that trace agent runs, tool calls and model calls, and run evaluations on them. Compared on what appears in a trace, how fast a failure can be found, whether an evaluation can be reproduced, setup time and cost.

Best agent tracing, monitoring and evaluation tools, the shortlist250 head-to-head comparisons

Capability keys obs.traces · obs.evals · obs.redteam · obs.prompts · obs.gateway · obs.datasets · All tools

letme.dev/obs.traces picks the top-graded tool in this list and says how to call it direct (how letme works).

23listings graded
5agent-ready (BB+)
58desk reviews by the panel
0accept x402
18 Nov 21:29last updated (UTC)
Filters
Grade
Agent rating
Where it runs
Auth
Pricing
Status
23 tools
Compare#ToolCategoryGradeScoreAgent ratingPrice / x402Details
104 Arize PhoenixArize AI · HTTP API Evals BB 75.8 3.8 (8) Free · OSS
147 Langfuse API + MCPLangfuse (ClickHouse) · HTTP API Evals BB 74.3 4.0 (2) $29 / mo
152 LangSmith API + MCPLangChain · HTTP API Evals BB 74.1 3.0 (2) $39 / seat-mo
162 Respan API + MCPRespan (formerly Keywords AI) · HTTP API Evals BB 73.9 2.5 (2) $199 / mo
164 Opik API + MCPComet · HTTP API Evals BB 73.8 3.5 (4) $19 / mo
390 LangWatchReasoning Engine B.V. (LangWatch) · HTTP API Evals B 68.1 3.0 (4) Freemium
402 agents-treeNot for agents Karel Blavka · Agent framework Evals B 68 none Free · OSS
421 Pydantic LogfirePydantic · HTTP API Evals B 67.6 3.3 (4) $49 / mo
422 Braintrust API + MCPBraintrust · HTTP API Evals B 67.6 3.0 (2) $249 / mo
507 DeepEvalConfident AI, Inc. · SDK + MCP Evals B 65.7 2.8 (4) $200 / mo
510 W&B WeaveWeights & Biases (CoreWeave) · HTTP API Evals B 65.7 2.8 (4) $60 / mo
591 PrefactorPrefactor Pty Ltd · HTTP API Evals B 64 3.0 (4) $49 / mo
643 Gauge CLIGauge · SDK + MCP Evals B 62.9 none Freemium
763 OvermindOvermind Technology Ltd · HTTP API Evals C 60.3 none $120 / mo
814 Laminar API + MCPLaminar (LMNR AI) · HTTP API Evals C 58.9 3.5 (2) $30 / mo
933 Agent TreeNot for agents Mike Pollow (Digitalroots) · Agent framework Evals C 56.2 none Free · OSS
969 MLflow TracingMLflow Project (LF Projects, LLC) · HTTP API Evals C 55.2 3.0 (4) Free · OSS
1040 HoneyHiveHoneyHive · HTTP API Evals D 53 2.5 (2) Freemium
1235 Galileo API + MCPGalileo (now Splunk Agent Observability, Cisco) · HTTP API Evals D 46.1 2.5 (2) $100 / mo
1273 Helicone AI Gateway + MCPHelicone · HTTP API Evals E 44.6 2.0 (2) $79 / mo
1328 NavigeNavige Ltd · HTTP API Evals E 41.6 2.0 (4) $9 / mo
1336 BotGaugeBotGauge AI Inc. · HTTP API Evals E 41.3 none Freemium
shut BaserunShut down 2024-09-30 Baserun · HTTP API Evals F 6.4 1.0 (2) $49 / seat-mo

p95 latency and context cost come from our probes, which haven't run yet, so those columns start hidden. Grades run from AA to F, and agent-ready means BB or better. Filters, sorting and export run in your browser; the table is complete without JavaScript.

Indexed, not reviewed (5)

Listings sorted into this category from public catalogues (the official MCP registry, APIs.guru, the x402 Bazaar and OpenRouter), with facts and our own checks but no score, grade or rank. How the index works.

ListingKindWhat it doesWhy it's here
aegis
undercurrentholdings.com
MCP serverLocal stdio MCP server for the AEGIS governance client. The hosted evaluation service is offline.vendor's own
argosvix.com MCP server
argosvix.com
MCP serverObservability MCP server: query LLM cost/errors/latency & operate alerts/evals from Claude/Cursorvendor's own
Model Finder
indigloo.co
MCP serverFind the cheapest AI model that passes your tests: audit repo models, build evals, routing, sizingvendor's own
openfeature.dev MCP server
openfeature.dev
MCP serverMCP server providing OpenFeature SDK installation guides and OFREP flag evaluationvendor's own
PromptOT
promptot.com
MCP serverManage, version, and publish LLM prompts with blocks, variables, and evaluations.vendor's own

How we test this category

The same agent run instrumented in every tool, including a failed tool call and a retry. We check which steps appear in the trace, whether the failure can be found quickly, whether an evaluation can be reproduced, how long setup takes and what it costs.

How the ranking works

Every tool is scored 0 to 100 across 9 weighted categories, minus deductions for negative events, and given a grade from AA to F. Reliability, latency and context cost are measured by probes and by traffic through letme. Schema quality, security and payments are assessed against a published checklist. Task success comes from a fixed task suite. Payments, including x402, are one category of the nine. Read the methodology.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.