What an agent asks before it picks a tool, and what a site should answer

At half past twelve on Friday 6 November I asked what we're still missing to make Anchor Terminal more AI friendly, besides the tests I still need to start running, and what would help agents make better decisions about the tools they use. The list that came back had seven things on it. Reading it, I thought most of them were things any site or tool an agent uses to choose should do, whether it's a directory like ours, a registry, a vendor's docs or an MCP server, and we thought they were good things to have, so I picked three and we built them the same day.

We were already good at being a place an agent comes to look things up. There are 1,491 graded listings, every one with its score per category, the reasons, the sources, the prices and how to connect, as a page, Markdown, JSON, an MCP server and a DNS record ^[1]. An agent that's halfway through a job asks something harder than whether Resend is good. It wants to know whether the things it already runs are still the right ones, what loading this server is going to cost its context for the one thing it needs to do, and which of five tools wins on what its operator cares about, which might be security and might be the bill, and I think it almost never matches the weights anyone else picked.

Say "we don't know" when you don't

I think the first thing a site agents rely on should do is tell "no" apart from "we haven't checked", because an agent can't see the gap the way a person scrolling a page might.

We got this wrong on 15 October, when we added filters for what a company needs and what a first call takes, so an agent could search for an email API it can start using with no human, or one that's processed in the EU, or one that doesn't train on your data ^[2]. A listing whose answer we hadn't researched didn't match, and the answer didn't say that any had been left out for that reason. Search for "send email" with agentAlone=1 and you got one result out of 60 ^[3], and an agent reading that answer had no way to tell whether the other 59 need a human or whether we just hadn't looked (five of them need a human and we knew it, and we hadn't looked at the other 54). It was worse in the data than in the search. The short form of the research dropped agentAlone, freeTier and dpa from the JSON when they were false, so a listing we'd checked and found needs a human looked the same as a listing we'd never checked.

Every filter now answers yes, no or unknown for every listing, and the search says how many it left out as each, by filter. Ask with unknown=include and the unknown ones come back too, marked with the filters we can't answer for them, and the research carries setupChecked and complianceChecked, so a false is a known no. With 107 of 1,491 listings researched for this ^[1], the answer to most of these filters is still "unknown".

Check what the agent already runs

An agent already knows what it runs, because it's sitting in its MCP client config, its package.json, its go.mod or its requirements.txt. I think a site like ours should take that file as it is and say what's changed, the way npm audit does for packages, rather than make the agent look each tool up one at a time, which is what it had to do on ours until that day.

So an agent now sends the whole thing to POST /api/v1/stack (or check_stack over MCP, with the same fields) and gets back, for every tool we list, the grade, what the vendor's status page says right now, our last probe, the dated sunsets and breaking changes, any security advisories on the repository, the newest version against the one it pins, what our workers saw change since it last checked, and up to two graded listings that do the same job better.

curl -s https://www.anchorterminal.com/api/v1/stack -d '{
  "mcpConfig": {"mcpServers": {
    "exa": {"url": "https://mcp.exa.ai/mcp"},
    "resend": {"command": "npx", "args": ["-y", "resend-mcp@2.20.0"]}}},
  "packageJson": {"dependencies": {"@anthropic-ai/sdk": "^0.30.0"}},
  "since": "30d"}'

We ran a made-up stack like that one against the data from the morning of 6 November. The Anthropic SDK pinned at ^0.30.0 came back a major version behind the 1.6.0 on npm, with five dated changes on the Claude API listing, two of them shutdowns at the end of November (claude-sonnet-4-5-20250929 retires on 30 November 2026) ^[8]. The Exa server came back with two deprecation notices from October and Tavily as a better graded listing for the same job, 77.1 against 67.8 and most of the gap from reliability. The react it found in the same package.json came back under unmatched.

The matching is by the names a listing already goes by, its npm and PyPI packages, its Go module, its repository and its MCP registry name, then by the host of its endpoint and, failing that, its domain, which the answer marks so an agent knows it was a guess. A sunset only becomes an action within 90 days or once it's passed since the last check, and "better" means at least a point higher in the same category with a capability in common, since a tie isn't a reason to switch. Our own products are never suggested, the same rule letme follows.

A site that takes an agent's config should also assume there's a secret in it. Some hosted MCP servers take the API key in the URL (Tavily's is ?tavilyApiKey=) ^[10], so every URL loses its query before we read it, the answer can't repeat a key, and nothing anyone sends is kept.

Say what a tool costs in context, and load only what the job needs

A server with a hundred tools costs every request a hundred definitions, whether the agent needs one of them or all of them. The biggest graded lists we hold run past 50,000 tokens ^[1], and among the 1,999 MCP servers in the index the biggest is past 400,000 ^[9].

I think a tool should say what it costs before an agent loads it, and a site that grades tools should say which part of it a job needs. Part of that is on the people who build MCP servers, since a server that answers tools/list without a key lets anyone see what it costs, and one that marks what deletes or overwrites lets a client hold those back.

For a site like ours, GET /api/v1/toolset?slug=railway-mcp-server&job=read+the+logs+of+a+service (or scope_tools) picks the tools that job needs and says what they cost against the whole list. For Railway it's get-logs and list-services, about 1,900 tokens instead of about 52,000 ^[4]. It comes with the config that loads only those, an mcp_toolset entry for the Claude API that turns everything off and the two on ^[5], allowed_tools for OpenAI's Responses API ^[6], and Claude Code's permission rules (those decide what runs without asking, not what gets loaded, and the answer says so). For any server we don't hold a list for, which is most of them because they want a key first, the agent posts its own tools/list answer and gets the same back.

We could have used a model to pick the tools, and for now we don't. It would read the descriptions better than word matching does, and it would also turn the pick into something that costs a call, changes from one run to the next and can't explain itself. The pick is by words, the job's words against each tool's name, title and the first sentence of its description, with the common verbs folded together so "look up" finds get_ and "remove" finds delete_. A word that's only in a description doesn't get a tool picked, because descriptions are where tools name other tools. Anything that deletes, revokes or overwrites is held back unless the job asks for it, and the answer lists what it held and why. It's not perfect (ask Tavily's list to "read a page" and nothing matches, because the tool is called tavily_extract), and when nothing matches it hands back the names so the agent can ask again in the server's own words.

Rank by the agent's priorities, and say why

Every ranking has someone's weights in it. Ours weighs reliability at 16, security at 14, schema and ergonomics at 13, payments at 10, and maintenance and transparency at 7 (performance and task success are 10 each and waiting on the probes) ^[7], picked for an average agent that doesn't exist. I think a site that ranks tools should publish its weights, let the agent bring its own, and say in a line why one result is above the next.

The search now takes weights=security:3,payments:0 (an object in search_tools), multiplies our weights by the agent's, and ranks by what we call yourScore, the same formula with the agent's numbers in it. With every multiplier at 1 it reproduces all 1,491 published scores to the decimal, and we checked that before anything else, because a re-ranking that drifts from the published number for no reason would be worse than none. The published score stays in the answer next to it.

For "send email", our order is Nylas at 79.1, Resend at 78.1 and AhaSend at 75.4 ^[3]. Triple security and ignore payments, and AhaSend goes up to second at 79.2 and Resend drops to third at 78.3, and Twilio SendGrid comes up from below MailerSend. Every graded result now carries a line saying why it's above the next one, so the agent doesn't have to diff two score tables to find out.

nylas-email   85.9  Ahead of AhaSend by 6.7 points under your weights, most from Security & auth (89 against 78, weighted ×3)
ahasend       79.2  Ahead of Resend by 0.9 points under your weights, most from Security & auth (78 against 69, weighted ×3)
resend        78.3  Ahead of Twilio SendGrid by 2.1 points under your weights, most from Reliability (90 against 76)

Weighting performance and task success does nothing yet, because nothing has a score for them until the probes and the task suites run on the box, and the answer says that too.

What's still not done

Most of this is only as good as how much we've researched, and that's the part that's thin. We hold whole tool lists for 19 graded servers and 449 indexed ones (another 415 indexed servers want a key before they'll list anything) ^[9], the requirement filters cover 107 listings, and the advisories are the ones published on a repository, not matched to the version an agent pins (that's the OSV poller, which is on the Svalbard list). Two more things on the list of seven were built later that evening, letme's pick taking the same weights, and the listings saying what our pollers see now that the benchmark's probes haven't run.

The tests on the box are still the thing I need to start, and when they run, performance and task success get scores and the weights on them start meaning something. Until then, if you run an agent, send it your config and see what comes back, and if you build a site or a tool agents choose from, I'd start with saying "we don't know" where you don't. If we get something wrong, file an issue and tell us.

References

Read on 6 November 2026.

1. [1]Anchor Terminal, 2026. The directory as JSON, every graded listing with its scores, requirements and live record, read on the morning of 6 November 2026. The source for the count of graded listings, the 107 with setup or compliance researched, the tool lists held (19 complete, 69 asking for credentials) and their sizes in tokens. anchorterminal.com
2. [2]Anchor Terminal changelog, 15 October 2026, "what it takes to start, and what a company needs to know". The source for when the requirement filters went in. anchorterminal.com
3. [3]Anchor Terminal, 2026. The search API, asked for "send email" with agentAlone=1 on 6 November 2026 before this change (one result of 60 matches), and without the filter for the published order and scores. anchorterminal.com
4. [4]Anchor Terminal, 2026. The Railway MCP server listing, with the 71 tools its hosted endpoint listed when our tracker last read it on 6 November 2026. The source for the tool names and the token counts for the whole list and for the two picked. anchorterminal.com
5. [5]Anthropic, 2026. Claude Platform docs, MCP connector. The source for the mcp_toolset entry, with default_config turning every tool off and configs turning the chosen ones on. platform.claude.com
6. [6]OpenAI Cookbook, 2025. Guide to using the Responses API's MCP tool. The source for allowed_tools limiting which of a server's tools are loaded. developers.openai.com
7. [7]Anchor Terminal, 2026. The benchmark, methodology v0.7, with each category's weight and the formula for the total. The source for the weights. anchorterminal.com
8. [8]Anchor Terminal, 2026. Sunsets, the dated shutdowns, breaking changes and price changes, read on 6 November 2026. The source for the Claude API's changes and the retirement of claude-sonnet-4-5-20250929 on 30 November 2026. anchorterminal.com
9. [9]Anchor Terminal, 2026. The index of listings built from public catalogues, each indexed MCP server's JSON read on 6 November 2026 for the tool list its hosted endpoint answered with. The source for the counts of indexed tool lists and the largest. anchorterminal.com
10. [10]Anchor Terminal, 2026. The Tavily API and MCP listing, whose hosted MCP address takes the API key as a query parameter. anchorterminal.com