# We test a model on our own site before it tests anyone else's tools

> Vlad Cealicu on how Anchor Terminal's lab tests a model before it tests anyone else's tools. A model gets 59 questions about our own site and has to look each answer up. For 46 of them the right answer is worked out when the question is asked, every model does the set three times, and the same model scored 55 and 64 of 66 on two passes over the same questions.

- Canonical: https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first
- Markdown: https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first.md (~2,900 tokens)
- Slim: https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first.min.md (~1,230 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-22

By Vlad Cealicu*, 22 November 2026. 1,514 words, about 7 min to read. * With the editing help of Claude Opus or Claude Fable.

- 59: questions about Anchor Terminal, answered through our API, our pages and our MCP server
- 46: of them have a right answer that's worked out from the site when the question is asked
- 5: are traps, four things that don't exist and one tool the question says to leave alone
- 3: passes for every model, and a question counts when it's right in more than half of them

Anchor Terminal's lab harness, its notes [1] and its question list [2], read 22 November 2026.

The lab that tests the tools we list had a bench before it had an evaluation. The bench asks whether a model can get each tool on a server to answer, so it could tell me which models were able to drive a tool, and it said nothing about whether they came back with the right answer [[1]](#ref-1).

I need to know that because the model doing this work has to run on the lab box. The lab tests other people's servers, the transcripts hold their tool descriptions and a model's words about them, and those stay on the box, so the harness refuses a model whose address isn't that machine [[1]](#ref-1). That leaves me the open-weight models that fit in 128 GB of memory, and I wanted a number to choose between them with. The scores are in [the other post](/blog/local-models-on-one-box-against-claude).

## You can test an agent on your own product

To mark an answer right you have to know the answer, on a real server, on the day you ask, and the one site where we know every right answer is our own [[1]](#ref-1). So the evaluation is 59 questions an agent might ask Anchor Terminal [[2]](#ref-2). What grade do we give a listing, which of two model APIs has the higher score, how many listings in a category have a reliability score of 90 or more, what share of MCP servers are at or above the grade we call agent-ready.

I think anyone with an API can do the same, and should before an agent's mistakes with it reach a customer. You already know your own right answers, so the questions are quick to write. What you learn is which models can use what you built, and where they trip. Our harness is public under the MIT licence for that reason, and a vendor can run it against their own server and check our numbers [[1]](#ref-1).

## A test nobody can learn by heart

One question, from the asking to the mark

1. **The right answer** (the harness). Worked out from our public API as the site is now, and kept from the model
2. **The question** (the harness). Sent to the model with the tools it may call and the limits it has
3. **The lookups** (the model). Fetches of our API and pages and calls to our MCP server, made directly or from a Go program in a cell with no network
4. **The answer** (the model). One line, within 32 replies, 200 tool calls and 200 fetches
5. **The mark** (the harness). The answer checked against the site as it was when the question was asked and as it is now, and against the lookups the model made

For 46 of the 59 the right answer isn't written down anywhere [[2]](#ref-2). The harness works it out from our public API when the question is asked, and checks the model's answer against the site as it was then and as it is when the answer comes back, so a listing we re-grade in the middle doesn't cost a model the question [[1]](#ref-1). Eight more have answers that don't move, such as the highest grade on our scale, and the last five are traps [[2]](#ref-2).

That's what lets the questions be public. A model can't learn answers that change with the site, so I don't have to hide the test to keep it honest, and anyone can read what we ask [[1]](#ref-1).

## What counts as a right answer

An answer with no lookup behind it counts as wrong even when it's right [[1]](#ref-1). The test is about what an agent does and not what it knows. Prices, grades and tool lists change, and an agent that answers from memory is wrong about them the day they do.

Four of the traps ask about something that isn't there (a listing, a model on a price list, a category), and the right answer is NONE. The way to find that out is to ask and be told so, with a 404 from our API or an error from a tool, and being told counts as the lookup [[4]](#ref-4). The fifth trap asks which of our tools submits a signed review and tells the model not to call it, and one call to that tool makes the answer wrong whatever the model says next [[2]](#ref-2).

Every model does the set three times, and a question counts when it's right in more than half of the passes it was scored in [[1]](#ref-1). The Lab score is the questions right out of the questions scored, with a level 2 question counting twice and a level 3 question three times, so the 12 hardest questions are 36 of the 110 points [[1]](#ref-1).

Three passes matter more than I expected. On the harder set further down, Claude Haiku 5.5 got 64, 62 and 55 of the same 66 questions in three passes, each a new instance with the same instructions [[1]](#ref-1). That's nine questions between two runs of one model, so when somebody shows me an agent benchmark with one run a model, I ask how far a second run would have moved it.

## What the harness gives a model

A score for an agent is a score for the model and for the harness around it, so I want the harness to stay out of the way. The limits it keeps are the ones that protect somebody else's server (a pause between requests, one address a run, no call to a tool that changes something), and the rest are there so a model can do its best at testing the tools we list [[3]](#ref-3).

A model is told its limits at the start. A question gets 32 replies, 200 tool calls and 200 fetches, a reply can be 16,384 tokens, and a model's context is as long as the box has memory for, up to 131,072 tokens [[3]](#ref-3). It can call tools itself or write a Go program that calls them, as it chooses, and the program runs in a cell with no network and no files [[1]](#ref-1). A program that fails to build only because of an unused import is fixed and built again without costing a reply [[3]](#ref-3). When a question is lost, the harness says how, whether the model ran out of replies, gave no answer, called a tool it was told to leave alone or answered with nothing looked up [[3]](#ref-3).

## A model the box can't run

A hosted model takes the same test by hand. The harness prints what a model on the box is given, writes the right answers to a file without showing them, and later grades a file of answers with the same code [[1]](#ref-1). Every pass is a new instance with no history and one logged way to reach the site, and what counts as a lookup comes from that log and never from the model's own account [[1]](#ref-1). A pass is one conversation for the whole set, where a model on the box starts each question fresh, so a score made this way is printed with a note that it was graded outside the harness and is never ranked beside the box's [[1]](#ref-1).

## What's hard for an agent is the plan

A second set of 66 questions is kept beside the normal one, for when the normal one stops telling models apart. It never makes a Lab score and isn't ranked [[1]](#ref-1).

Its questions are long jobs. Some need one field from each of 60 or 80 or 177 listings, and the quick way is to notice that our search filters on the same field and says how many match [[1]](#ref-1). Six are work with something to hand in, such as a stack of four listings that keeps four rules and comes within three points of the best total any such stack can reach, checked by a verifier because there are many right answers [[1]](#ref-1).

Every hard question is still about the same public data, with its answer worked out when it's asked [[1]](#ref-1). More knowledge wouldn't help a model with any of them. They take more steps, a plan that changes half way and something to hand in at the end, and I think that's where models will differ from here, more than in what they know.

## What's next

Every model runs again on the harness as it is now, and the best of them take the harder set, which no model on the box has taken.

After that the lab tests plain HTTP APIs. Over half of what we list is APIs [[1]](#ref-1), and I don't think MCP is where agents end up. What an agent does with a service is write a small integration against its API to get what it needs out of it, so that's what the lab will ask a model to do, from what the API says about itself [[5]](#ref-5).

---

More in [Inside Anchor Terminal](https://www.anchorterminal.com/blog/inside/index.md) · Previous [How to build an agent-first website, from our first six weeks and 5 million page views](https://www.anchorterminal.com/blog/how-to-build-an-agent-first-website.md)

- Older: [The Log, issue 7: the IPO, the price war, the blame](https://www.anchorterminal.com/blog/the-log/issue-7-the-ipo-the-price-war-the-blame.md)
- Newer: [An open model on a £3,879 box scored 98 on our test. Claude scored 100](https://www.anchorterminal.com/blog/local-models-on-one-box-against-claude.md)
- All posts: https://www.anchorterminal.com/blog/index.md

Anchor Terminal grades the tools AI agents run on, with a source for every score. [The top tools](https://www.anchorterminal.com/top/index.md) · [Categories](https://www.anchorterminal.com/categories/index.md) · [Compare two tools](https://www.anchorterminal.com/compare/index.md) · [Prices](https://www.anchorterminal.com/prices/index.md) · [Sunsets](https://www.anchorterminal.com/sunsets/index.md) · [API, MCP server and CLI](https://www.anchorterminal.com/agents/index.md)

## References

Read on 22 November 2026.

[1] Anchor Terminal, anchorlab, README.md and DESIGN.md. Source for the bench and the evaluation, the model having to be on the box, how a question is asked, marked and scored, runs by hand and their scores, and the hard set. https://github.com/Anchor-Terminal/anchorlab

[2] Anchor Terminal, anchorlab, eval/tasks.go. Source for the 59 questions, the 46 whose answers are worked out when asked, the eight whose answers don't move and the five traps. https://github.com/Anchor-Terminal/anchorlab/blob/main/eval/tasks.go

[3] Anchor Terminal, anchorlab, pull request 35. Source for the limits a question has and the model being told them, the tokens and context it gets, the repaired build, the reasons a question was lost and which limits stay. https://github.com/Anchor-Terminal/anchorlab/pull/35

[4] Anchor Terminal, anchorlab, pull request 34. Source for being told a thing isn't there counting as the lookup on the four NONE traps. https://github.com/Anchor-Terminal/anchorlab/pull/34

[5] Anchor Terminal, anchorlab, pull request 37. Source for the lab testing a plain HTTP API from its OpenAPI document. https://github.com/Anchor-Terminal/anchorlab/pull/37

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
