# We test a model on our own site before it tests anyone else's tools (slim)

> Vlad Cealicu on how Anchor Terminal's lab tests a model before it tests anyone else's tools. A model gets 59 questions about our own site and has to look each answer up. For 46 of them the right answer is worked out when the question is asked, every model does the set three times, and the same model scored 55 and 64 of 66 on two passes over the same questions.

- Full: https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first.md (~2,900 tokens) · this version ~1,230 tokens · JSON https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first.json · canonical https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first
- Index: https://www.anchorterminal.com/llms.txt · API: https://www.anchorterminal.com/api/v1/index.json · MCP: https://www.anchorterminal.com/mcp · Updated: 2026-11-22
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)

By Vlad Cealicu*, 22 November 2026. 1,514 words, about 7 min to read.

- 59: questions about Anchor Terminal, answered through our API, our pages and our MCP server
- 46: of them have a right answer that's worked out from the site when the question is asked
- 5: are traps, four things that don't exist and one tool the question says to leave alone
- 3: passes for every model, and a question counts when it's right in more than half of them

Anchor Terminal's lab harness, its notes [1] and its question list [2], read 22 November 2026.

The lab that tests the tools we list had a bench before it had an evaluation.

I need to know that because the model doing this work has to run on the lab box.

## You can test an agent on your own product

To mark an answer right you have to know the answer, on a real server, on the day you ask, and the one site where we know every right answer is our own [[1]](#ref-1).

I think anyone with an API can do the same, and should before an agent's mistakes with it reach a customer.

## A test nobody can learn by heart

One question, from the asking to the mark

1. **The right answer** (the harness). Worked out from our public API as the site is now, and kept from the model
2. **The question** (the harness). Sent to the model with the tools it may call and the limits it has
3. **The lookups** (the model). Fetches of our API and pages and calls to our MCP server, made directly or from a Go program in a cell with no network
4. **The answer** (the model). One line, within 32 replies, 200 tool calls and 200 fetches
5. **The mark** (the harness). The answer checked against the site as it was when the question was asked and as it is now, and against the lookups the model made

For 46 of the 59 the right answer isn't written down anywhere [[2]](#ref-2).

That's what lets the questions be public.

## What counts as a right answer

An answer with no lookup behind it counts as wrong even when it's right [[1]](#ref-1).

Four of the traps ask about something that isn't there (a listing, a model on a price list, a category), and the right answer is NONE.

Every model does the set three times, and a question counts when it's right in more than half of the passes it was scored in [[1]](#ref-1).

Three passes matter more than I expected.

## What the harness gives a model

A score for an agent is a score for the model and for the harness around it, so I want the harness to stay out of the way.

A model is told its limits at the start.

## A model the box can't run

A hosted model takes the same test by hand.

## What's hard for an agent is the plan

A second set of 66 questions is kept beside the normal one, for when the normal one stops telling models apart.

Its questions are long jobs.

Every hard question is still about the same public data, with its answer worked out when it's asked [[1]](#ref-1).

## What's next

Every model runs again on the harness as it is now, and the best of them take the harder set, which no model on the box has taken.

After that the lab tests plain HTTP APIs.

---

More in [Inside Anchor Terminal](https://www.anchorterminal.com/blog/inside/index.md) · Previous [How to build an agent-first website, from our first six weeks and 5 million page views](https://www.anchorterminal.com/blog/how-to-build-an-agent-first-website.md)

- Older: [The Log, issue 7: the IPO, the price war, the blame](https://www.anchorterminal.com/blog/the-log/issue-7-the-ipo-the-price-war-the-blame.md)
- Newer: [An open model on a £3,879 box scored 98 on our test. Claude scored 100](https://www.anchorterminal.com/blog/local-models-on-one-box-against-claude.md)
- All posts: https://www.anchorterminal.com/blog/index.md

Anchor Terminal grades the tools AI agents run on, with a source for every score.

## References

Read on 22 November 2026.

[1] Anchor Terminal, anchorlab, README.md and DESIGN.md.

[2] Anchor Terminal, anchorlab, eval/tasks.go.

[3] Anchor Terminal, anchorlab, pull request 35. Source for the limits a question has and the model being told them, the tokens and context it gets, the repaired build, the reasons a question was lost and which limits stay.

[4] Anchor Terminal, anchorlab, pull request 34. Source for being told a thing isn't there counting as the lookup on the four NONE traps.

[5] Anchor Terminal, anchorlab, pull request 37. Source for the lab testing a plain HTTP API from its OpenAPI document.
