The Agent Tool Benchmark

Why another benchmark

The directories that exist rank agent tools by GitHub stars, package downloads, or how often a tool gets called through one particular proxy. Those numbers say what is popular. They don't say whether the tool answers at three in the morning, whether its descriptions make sense to a model, how many tokens it burns before the first useful call, whether an operator can hand it to an agent without handing over the keys, or whether an agent can pay for it without a human. Popularity and quality have already come apart. The most-used documentation server in the MCP world ships a 2,000-character tool description that a community grader marked as failing (we checked, and it is 2,006 characters).

We spent a decade grading crypto exchanges with a public method, and the thing we learned is that once the checklist is public, the people being graded start fixing the items on it. So the plan here is the same. Assess what can be read against a published checklist and show the working, probe what can be probed, and say plainly when something couldn't be verified.

An agent doesn't run on MCP servers alone. It runs on a model, inside a framework, reading data from providers and pages it scrapes, and sometimes paying for what it uses. So the benchmark covers all of those, on one scale, with each category read the way that makes sense for the kind of thing being scored.

What we benchmark

Four kinds of listing share the nine categories. The weights don't change between kinds. What each category measures does, and this table is the whole mapping.

MCP servers, HTTP APIs, model APIs, frameworks, data providers and scraping tools are ranked together, because an agent builder does choose between them for the same budget. Payment protocols are graded on the same scale but not ranked against tools. You choose a protocol by who you're paying, not by its score, and putting a specification at number two above a search API told nobody anything useful. Retired listings keep their page and their grade and drop out of the ranking.

Categories and weights

Nine categories add up to 100 points. The weights follow what breaks agents in practice. A tool that is down is useless whatever its schema looks like, so reliability carries the most. A tool an agent can't pay for is a tool it can't use alone, so payments carry more than a payments category normally would. We're not certain the weights are right (they're our first guess, and the first run with probes will tell us where they're wrong).

Two of the nine are pending in this run. The "This run" column is the share of the 100 points each category carries until they're scored.

This run is each category's share of the 100 points while Performance and Task success are pending, its weight ÷ 80 × 100. A total is Σ(score × weight) ÷ 80 over the assessed categories, then the negative events.

How this run was made

Until 1 October 2026 every score on this site was an invented sample, published to show the method. The October 2026 research run replaced them.

Between 30 September and 2 October 2026, research agents running on Anthropic's Claude models researched every graded listing from public evidence. They read status pages and their incident history, rate-limit and error documentation, API references and OpenAPI files, MCP tool definitions in the source, pricing pages, terms and privacy policies, security and trust pages, changelogs, release tags and CI in cloned repositories, package registries and public advisories. They worked from the links on each listing and the pages those linked to, with a budget of about twelve page fetches per listing and no web search. Each category was scored against the checklist below. The points add up to the score, and the note beside each score on the listing says what earned them and where the agent departed from the checklist and why.

Five rules held for every listing.

• Score from evidence found. Nothing gets points for what a vendor probably does.
• Couldn't check isn't the same as absent. If a thing was looked for and isn't there (no status page, no SLA, pricing behind a login), it scores as absent. If a page couldn't be loaded, the agent could rely on the listing's own facts from their 30 September check and say so in the note. If those didn't cover it either, the item scores as absent, goes on the listing as an open question ("unchecked" and what), and the confidence drops. Our own fetch limits never read as a vendor's failing without saying so.
• Vendor claims are claims. "Exa says its index covers 1.4 trillion URLs" is a fact about Exa's page, not our measurement.
• No invented numbers, dates, incidents or quotes. If it wasn't found, it isn't true for this run.
• Facts on a listing that turned out wrong (a renamed product, a moved endpoint, dropped x402 support) were corrected, and the correction is named in the listing's open questions.

Every graded listing carries three things beside its scores. A confidence, high, medium or low, which is how sure we are that the scores would survive the vendor reading the notes. The open questions, under "What we couldn't check". And the sources, at least four per listing, each with the date it was read. The JSON for a listing has all of it under anchor.assessment.

The conflict rule. The research agents and the review panel run on Claude, so Anthropic is a company we depend on. Anthropic's listings (the Anthropic API and the Claude Agent SDK) were graded by the same checklist as every other listing, with each judgement call written in the note, and each carries a disclosure. The panel doesn't review them. Our own products are graded by a stricter version of the same rule (below). A listing the founder built somewhere else, the CoinDesk Data API, is graded like the rest and carries a disclosure too.

The checklist

Each category has a checklist. The points are what an item is worth, and a listing's score in the category is the sum. Read each one the way that fits the kind of listing (the table above). These are the checklists the research agents used, written for a reader.

Reliability, from public evidence

Hosted APIs, MCP servers, models and platforms.

• 20, a public status page with component history (Statuspage, Instatus, BetterStack or the vendor's own).
• 0 to 30, the incident record for the last 90 days on that page. 30 for a clean record or trivial incidents only, 20 for minor incidents only, 10 for one major outage (an hour or more of a core API down, or errors across the board), 0 for several. 5 when there's no history we could read, and the note says so.
• 15, rate limits documented with numbers.
• 15, documented 429 or overload handling (Retry-After, backoff guidance), and idempotency keys or safe-retry guidance where writes are involved.
• 10, an SLA published for any paid tier.
• 10, the surface agents use is generally available, not beta or preview.

Local packages, SDKs, frameworks and stdio MCP servers.

• 20, installs from an official package with supported runtimes stated.
• 25, a public CI and test suite, passing on the default branch.
• 0 to 25, open crash or regression issues relative to activity (25 for few and handled, 0 for many, old and unanswered).
• 15, semver discipline and breaking changes called out in a changelog.
• 15, version 1.0 or later, or declared stable.

Protocols are read from their reference implementations, the public facilitators or servers, spec stability and test vectors.

Schema and documentation

APIs and MCP servers.

• 25, a machine-readable contract (a public OpenAPI file or similar; for MCP, typed JSON Schema inputs on every tool).
• 10, llms.txt or Markdown docs served for agents.
• 0 to 20, descriptions that say what a tool is for, when to use it and when not to, read from the tool definitions in the source or the API reference.
• 0 to 15, typed inputs with enums, constraints and required fields, and no free-form JSON blobs.
• 0 to 15, examples and documented error responses.
• 15, versioning and a public changelog.

Models are read from the API reference, the OpenAPI file, llms.txt, the structured-output and tool-use docs and the model cards. Frameworks from docs a model can follow, typed interfaces, examples and the API reference.

Agent ergonomics

• 0 to 25, context cost. For MCP, the number and size of the tool definitions (25 for ten or fewer compact tools, 15 for 11 to 30, 5 for more than 30, plus up to 10 back for toolsets, dynamic loading or read-only subsets). For APIs, whether responses can be sized (field selection, limits, summaries).
• 20, pagination, filtering and output-size controls.
• 20, actionable, documented error responses, codes and messages an agent can recover from.
• 20, idempotency or safe retries, and for MCP the readOnlyHint and destructiveHint annotations.
• 15, sensible defaults, few required parameters, and official SDKs in at least two languages.

Models are read for tool use, structured output, prompt caching, context length, batch and SDKs. Frameworks for how much code and how many defaults a tool-calling agent with MCP needs.

Security and auth

• 0 to 30, the credential model. 30 for OAuth 2.1 with scopes, or scoped and revocable keys with rotation. 20 for plain revocable API keys. 10 for one all-powerful key. 10 off when a secret can travel in a URL query string as a documented option.
• 0 to 20, read-only or least-privilege modes, and confirmation or approval for destructive actions.
• 0 to 15, prompt-injection posture where the tool returns untrusted content (documented mitigations or guidance). A tool that returns no untrusted content gets 10.
• 0 to 15, audit logs or per-call visibility for the operator.
• 0 to 20, a security programme. security.txt or a disclosure policy, a bug bounty, SOC 2 or ISO 27001, advisories handled in public.

Models are read for retention, whether API data trains models (and whether that's off by default), zero-retention options and certifications. Frameworks for telemetry defaults, approval hooks, guardrails and sandboxing.

Payments and pricing

The published rubric, also on the x402 page.

• 40, a machine payment protocol (x402, MPP or L402) on the tool's own endpoints. 10 to 30 when it covers only some endpoints or only goes through a third party, and the note says which.
• 20, per-call or per-unit pricing published without a login. 10 for public plan-only pricing, 0 for "contact sales" or prices behind a login.
• 20, a free tier or trial that doesn't need a card.
• 20, autonomous onboarding, meaning an agent can get access without a person signing up in a browser (keyless use, x402, a programmatic key API).

Payment platforms and agent wallets rarely charge for their own API over a machine protocol, so the first line has steps for them, and the highest one that applies counts. 40 when x402, MPP or L402 runs on all their own endpoints, 30 when it runs on part of their own API, 25 when their merchants can accept one, 20 for running a facilitator, 15 for paying as a buyer, and 0 when the only protocol is their own. Merchant acceptance sits above a facilitator because the platform's own customers can charge agents through it, while a facilitator settles for sellers who wire up the protocol themselves. The counter-argument (a facilitator does more for the protocol as a whole) has a point. Each note says which step applied.

Open-source software you run yourself is scored on its hosted or paid option if it has one. A free, self-hosted package with nothing to buy gets 20, 20 and 20 for the last three lines, and 0 to 40 for the first only if it ships a payment protocol.

Maintenance and community

• 0 to 30, time since the last release, or the last published model or API change for a closed service. 30 within 30 days, 20 within 90, 10 within 180, 0 older.
• 20, at least three releases or dated changelog entries in the last 90 days.
• 0 to 25, responsiveness. Issues and pull requests answered on GitHub (the open issues and how recent the replies are). For closed services, a public changelog and a support or community channel that answers, 0 to 15.
• 15, presence in the official MCP registry under a verified namespace (MCP servers), or current official SDKs (APIs and models).
• 10, package health, current dependencies and CI.

Models are read for deprecation notice periods and model churn rather than release counts.

Transparency and trust, the editorial half

• 0 to 30, source availability and licence clarity. 30 for open source under an OSI licence, 15 for closed with clear terms, 0 for unclear terms.
• 0 to 30, data handling and retention statements that agree with each other (privacy policy, DPA, retention periods, subprocessors).
• 0 to 20, a deprecation policy or notices with dates.
• 0 to 20, telemetry disclosed with an opt-out (local software), or subprocessors and data locations disclosed (hosted).

The other half of Transparency and trust is the provenance score, computed from checked facts (below). The category score is the mean of the two.

Pending categories

Performance (weight 10) is latency per call, measured by our probes. Task success (weight 10) is a fixed task suite run through every tool in a category. Neither can be read from a vendor's pages, and neither has run, so this run doesn't score them. They show as pending everywhere a score is shown, with no number, and they add nothing to the total.

The total is the weighted mean over the seven assessed categories, renormalised so the maximum is still 100.

score = Σ (category score × weight) ÷ 80, over the seven assessed categories
        then the negative events, 0 to −15

80 is the sum of the assessed weights. So Reliability counts for 20 points in this run rather than 16, Security for 17.5 rather than 14, and so on (the "This run" column above). Grade bands are unchanged. One consequence is worth saying out loud. A tool that would do badly on latency or on real tasks isn't marked down for it yet, and a fast one isn't rewarded. Treat this run's grades as a reading of what a vendor publishes and how it runs its service, which is a lot, but not all, of what an agent meets.

What changes when the probes run

• Performance gets scored from p50 and p95 latency per call, measured from London, Virginia and Singapore.
• Reliability adds measured availability and error rates to the status history it's read from today.
• Context cost in Agent ergonomics becomes the measured token count of tools/list, in the default configuration and with every toolset on.
• Task success gets scored from the task suites, and for data providers half of it becomes the data-quality score already published on their listings.
• The weights go back to the published ones, and every listing's history records the change with the methodology version that made it.

Grades will move when that happens, some of them a lot. The probes and pollers that watch uptime today feed the live panel on each listing and don't change a score until a run.

Our own products

letme (letme.dev) is Anchor Terminal's own product, and LocalGhost is built by its founder. Until 2 October 2026 we listed them and didn't grade them. On 2 October both were graded by the same checklist as every listing, with four differences, because a grade we give ourselves has to hold up for a reader who assumes we were kind to ourselves. letme's grade still comes from that rule.

• Two research agents graded each one independently, and every judgement call was read strictly. Only what was live and public that day counted. Anything specified, planned or "coming" earned nothing, and so did our own description of how something works without the code, the live answers or a dated document behind it.
• A third agent, which graded neither, reconciled the two item by item on the checklist. The lower award stood unless it rested on an error the auditor checked for itself, and a deduction counted only if the auditor could still see the problem that day.
• The review panel doesn't review them.
• letme never picks them and never answers with them, so it can't favour the company that runs it. Their records say own: true. They're in rankings, categories and comparisons like any listing.

Since 3 October, at the founder's request, LocalGhost is graded the same way as every listing instead, with the same readings, neither stricter nor looser, the way a competitor of ours is. Two research agents grade it independently and a third reconciles them, checking the evidence itself wherever they disagree, with no award standing by default. The panel still doesn't review it and letme still never picks it, because those protect against us favouring ourselves rather than marking us down. letme moves to the same rule at its next check.

Each carries a disclosure that says how it was graded, the reason and sources for every score are on the listing like any other, and anyone can dispute an item the same way. Every grading and reconciliation is kept with the research run.

Listed, not graded

A listing can be listed and not graded (graded: false), with no score, grade, rank, reviews or letme pick, and a disclosure that says why. None is today.

Competitors of our own products

When a listing competes directly with one of our own products, a reader could suspect the founder's side marked it down, or that we went easy on it to look fair. So it's graded by the same checklist as every listing, neither stricter nor looser, and the grading is set up so that neither can creep in.

• Two research agents grade it independently. Where a checklist line could be read two ways, they take the reading already used for the same line on comparable listings. They don't look at our own product's listing and don't compare the two.
• A third agent, which graded neither, reconciles them item by item. Where the graders agree, their award stands unless the auditor finds it rests on an error. Where they disagree, neither award stands by default. The auditor checks the evidence itself and awards what it supports.
• What we couldn't read counts as absent, as for every listing, and the note says when a score is low because we couldn't read something rather than because it's missing. When a vendor's site refuses our reader, text supplied to us from it counts, and the listing says who supplied it.
• The review panel reviews it like any listing, and letme can pick it.
• A disclosure at the foot of its page, and of any comparison it's in, says which of our products it competes with and how it was graded. Its record says competesWith and names that product.

Four listings are graded this way today, all competing with LocalGhost. Underdog was the first, on 3 October 2026, and Khoj, Open WebUI and Screenpipe followed the same day, the three that LocalGhost's own about page names as its competitors. A listing is graded this way when our product names it as a competitor, or when it does the same job for the same people, as Underdog does. Both gradings and the reconciliation are kept with the research run for each.

Negative events

Scores describe the steady state. Negative events describe what happened. Up to 15 points come off for incidents in the last twelve months that can be sourced. A security incident or an exfiltration path, up to 15. A breaking change shipped without notice, 3 to 8. An endpoint removed while still advertised, 3 to 6. Telemetry nobody disclosed, 2 to 5. A misleading listing or claim, 2 to 5. Deductions decay, and a vendor that documents the fix and ships mitigations gets the deduction lifted early. A vendor that ships a breaking rename in a minor release and says nothing carries the full term. Every deduction is written on the tool's page with a date and a source.

Grades

Agent-ready means BB or better, which is our shorthand for "you can give this to an autonomous agent as long as you read the caveats on its page". Below BB expect to supervise it, patch around it, or replace it. F is for tools that fail the checklist across the board or can't be verified at all.

Who's behind it

Credibility shouldn't be a feeling. For every listing we record the legal entity named in its terms, its registrable domain and the date the registry says it was first registered, whether the hosted endpoint sits on a domain the vendor controls, and whether there are terms, a privacy policy, a status page, a changelog and a valid security.txt. Each check is worth fixed points and the total is the provenance score out of 100. It's half of Transparency and trust. The other half stays editorial, because licence clarity, retention statements that agree with each other and honest telemetry disclosure need reading, not counting.

Domain age is the weakest of these and we know it. A registration date is when the name was first taken, not when the current owner bought it, so slack.com (1992) and notion.com (1997) look older than the companies are. Every listing where that's true says so under the checks. That's why age is 15 points and never decides a grade on its own. A brand-new domain with a named company, published terms and a working status page still scores well, and a very old domain with nobody's name on it doesn't.

Registry dates come from each registry's own RDAP server. Where the registry doesn't publish one (.gov.uk, some brand top-level domains) we say so on the listing and score government domains in full.

Data quality

A clean API over thin data still fails the task. For data providers, half of Task success will be a data-quality score computed from what the data covers, how fresh and how deep it is, whether the method and sources are published, what the licence lets you do, and whether the publisher has official standing. Coverage is the one judgement call in it, rated 1 to 5 with the evidence written next to the number. Task success is pending in this run, so the data-quality score is computed and published on each data provider's listing, and it isn't in the total until the task suite runs. Each listing also says what its terms allow for caching and redistribution, because an agent that stores a price it wasn't allowed to store is a problem for its operator.

How each category is measured

In this run, from public evidence

Reliability, schema and documentation, agent ergonomics, security, payments, maintenance and the editorial half of transparency were scored against the checklists above by research agents reading what a model reads and what an operator would check. The tool definitions, the docs, the auth flow, the status history, the pricing, the terms and the repository. Each score's note is on the listing, with the sources. Vendors can dispute any item with evidence, and disputes are answered in public.

Machine payment in this run was read from each vendor's documentation, its source and its listing in the x402 Bazaar where it had one.

When the probes and task suites run

Remote tools will be probed every five minutes from London, Virginia and Singapore. A probe opens a session, lists tools, and runs one small fixed call that doesn't change state. Local tools will be started in a clean container every fifteen minutes, listed, and called once. We'll record availability, latency percentiles, error classes, timeouts, and whether the response shape matched the previous run. The token count of tools/list gets measured with a reference tokeniser in the default configuration and again with every optional toolset switched on, and both numbers are published, because the gap between them is the first thing to configure.

Machine payment will be detected, not read. For x402 we call the endpoint without paying and look for a 402 carrying a PAYMENT-REQUIRED header (v2) or a JSON body with x402Version: 1 (legacy) that names a scheme, a CAIP-2 network, an amount, an asset and a payee. MPP (WWW-Authenticate: Payment) and L402 (WWW-Authenticate: L402) challenges get detected the same way. For tools that settle, one real micropayment a week from a canary wallet, with the transaction hash published.

Each category will have a suite of representative tasks (find a fact on the web, open a pull request, run a read-only query, pull a table out of a page). Once a month the suite runs through every tool in the category with a fixed reference model and a fixed harness, and we record pass@1, pass@4, turns, tokens, and whether the tool's own error messages were enough to recover from a first failure. The suites will be published so vendors can run them.

Maintenance and community, by public signals

Release cadence, time since the last release, responsiveness on issues and pull requests, presence in the official MCP registry under a verified namespace, and dependency health, all read from public repositories and package registries on the run date. For model APIs this is mostly about retirements. How much notice, how often, and whether a retired id fails loudly. Every dated change we find goes on the listing and on Sunsets, with a calendar feed at /sunsets.ics.

Leaderboard

The full ranking with every assessed category score. Sort and filter in the directory, or fetch it as JSON at /api/v1/rankings.json.