{
  "data": {
    "author": "Vlad Cealicu",
    "faq": null,
    "kicker": "Blog · Inside · 9 November 2026",
    "lede": "Anchor Terminal is a benchmark of 1,651 listings that agents research, grade, review and publish, and two of us run it. This is how it's put together, why it's put together that way, and how we keep it sane.",
    "readingMinutes": 5,
    "words": 1155
  },
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/blog/a-machine-that-builds-the-benchmark",
    "json": "https://www.anchorterminal.com/blog/a-machine-that-builds-the-benchmark.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/blog/a-machine-that-builds-the-benchmark.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/blog/a-machine-that-builds-the-benchmark.min.md"
  },
  "markdown": "By Vlad Cealicu, 9 November 2026. 1,155 words, about 5 min to read.\n\nOn 9 November 2026 the benchmark had 1,651 listings, every one graded against a published checklist with the reasons and the sources for each score [[1]](#ref-1), and the repository behind it had 23 design specs, 142,723 lines of Go and 662 pull requests [[2]](#ref-2). Quynh and I started it on 25 September. Agents wrote nearly all of it, and every change went in through a pull request one of us read and merged.\n\n- 1,651: listings, one file each\n- 15: reviewers, all agents\n- 662: pull requests\n- 2: of us\n\nAnchor Terminal, 9 November 2026\n\nThe model matters, and the [two million page views post](/blog/what-the-next-model-is-for) says which one we use. I think the way the machine is laid out matters as much, because it's what lets an agent that knows nothing about us do a piece of the work and check it without one of us standing next to it.\n\n## The parts of the machine\n\nFrom a lead to a page\n\n1. **Research** (agent). One research agent per listing reads the rules and writes three files, the facts, a dossier with every score and its sources, and the people and investors behind it [[3]](#ref-3)\n2. **Land** (person). One script turns those files into the listing, run by whoever coordinates the batch [[3]](#ref-3)\n3. **Review** (agent). Eight panel reviewers and six audience reviewers each read it from their own angle, and an arbiter rules on what they wrote [[4]](#ref-4)\n4. **Watch** (worker). Workers inside the server call the endpoint every 5 minutes, read status pages every 10 and packages, security.txt and domains daily or weekly [[5]](#ref-5)\n5. **Publish** (build). The server renders the page with its Markdown and JSON twins, and only goes live if the checks pass [[5]](#ref-5)\n6. **Merge** (person). Quynh or I read the pull request and press the button [[6]](#ref-6)\n\nEach part does one job and hands the next part a file. A research agent never touches git or runs the site, it writes its three files and replies with a line per listing [[3]](#ref-3). A reviewer never changes a score. The workers watch the vendors and raise a change for an editor when what they see disagrees with the listing, rather than fixing the listing themselves [[5]](#ref-5).\n\n## Why it's laid out like this\n\nThere's no database behind the benchmark. A listing is one JSON file, a blog post is one Markdown file, the research behind a listing is a dossier next to it, and the rules for writing all of it are Markdown files at the top of the repository [[5]](#ref-5). An agent can read any of it with the tools it already has, every change is a diff we can read, and git keeps the history. The plan for Svalbard, the release where the site runs itself, says editorial data stays in files because diffs, provenance and the build's checks are worth more there than a database, and that everything a page says that can be computed is computed, so models do the research, the grading and the reviews and nothing else [[7]](#ref-7).\n\nThe site is one Go server written against the standard library, with two outside packages, both for checking the wallet signatures on people's reviews [[5]](#ref-5). The same server renders every page as HTML, Markdown, slim Markdown and JSON from the same files, so what an agent reads on the site and what an agent edits in the repository are the same thing. There's no framework and no front-end build, and I think that leaves an agent fewer things to get wrong (my guess, I haven't measured it).\n\n## How we keep it sane\n\nThe build checks itself. `go run ./gen -check` builds the whole site and fails on broken links, missing twins, pages that break our SEO rules, em dashes and banned wording in the prose, and blog references that are cited and missing or listed and never cited [[8]](#ref-8). The live server runs the same checks whenever the data changes, renders the new site into a scratch directory and swaps it in only if they pass, and if they don't the old site stays up [[5]](#ref-5). So the worst a bad change can do is not go live.\n\nThe reasons are written down. Every feature has a spec with what was decided, why, what we rejected and the numbers, and where I decided something in a chat the spec keeps my words [[9]](#ref-9). CLAUDE.md, the file a coding agent reads first, is a map of where things live and the rules for branches and pull requests [[6]](#ref-6). A model starts every session knowing nothing about us, and the specs are how a decision I took in October is still followed in November by an agent that wasn't there.\n\nThe rules are in one place. Until 8 November the research rules were a chain of batch files, each saying follow the previous one with these differences, and a research agent read about 36,000 tokens of old logs to find them, with some rules changed three times along the way. We folded them into one file of 167 lines, and when a decision changes a rule it changes there, in the same pull request [[3]](#ref-3).\n\nAgents check agents. For the riskier parts a second agent that hasn't seen the code reviews it, and for the security review of the merges from 6 to 8 November it was two reviewers and then a third agent that reproduced every finding itself before we believed it [[10]](#ref-10). The arbiter does the same for the review panel, reading every review against the evidence and marking it upheld, corrected or rejected [[4]](#ref-4).\n\nPeople decide. No agent pushes to main, every change is a pull request one of us merges [[6]](#ref-6), and our own products and their competitors are graded by the same published rules, with a disclosure on the page [[1]](#ref-1).\n\n## What doesn't work yet\n\nThe full test suite renders the whole site, takes about 15 minutes and needs several GB of memory, and a full build takes about 4 GB of disk, so a session on a small machine runs the checks after the pull request is open, not before [[6]](#ref-6). The spelling lint has been off since 4 November [[6]](#ref-6). The rule that everything goes through a pull request is a convention, since GitHub Free doesn't let a private repository protect main, so it holds because we and the agents keep it [[5]](#ref-5). All fifteen reviewers run on one family of models, Claude, where the design wants several, and the arbiter hasn't ruled on every listing yet [[4]](#ref-4). And two of the nine categories, Performance and Task success, are still pending on every listing until the measured runs start on our own box [[1]](#ref-1). The slow part is still us reading what the agents made.\n\n## You can do this too\n\nIf you're building something like this with agents, I'd start with the boring parts. Put the data in files, give each agent one job that ends in a file, have one command that builds everything and fails loudly, write each decision down with its reason where the agent will read it first, and make every change come through something a person merges.\n\n---\n\nMore in [Inside Anchor Terminal](https://www.anchorterminal.com/blog/inside/index.md) · Previous [Find the best tools for agents, and how we decide what to return](https://www.anchorterminal.com/blog/find-the-best-tools-for-agents.md)\n\n- Older: [The biggest gap between startups and enterprise we've ever had](https://www.anchorterminal.com/blog/the-biggest-gap-between-startups-and-enterprise.md)\n- All posts: https://www.anchorterminal.com/blog/index.md\n\nAnchor Terminal grades the tools AI agents run on, with a source for every score. [The top tools](https://www.anchorterminal.com/top/index.md) · [Categories](https://www.anchorterminal.com/categories/index.md) · [Compare two tools](https://www.anchorterminal.com/compare/index.md) · [Prices](https://www.anchorterminal.com/prices/index.md) · [Sunsets](https://www.anchorterminal.com/sunsets/index.md) · [API, MCP server and CLI](https://www.anchorterminal.com/agents/index.md)\n\n## References\n\nRead on 9 November 2026.\n\n[1] Anchor Terminal, the benchmark. Source for the checklist every listing is graded against, the reasons and sources for each score, the rule for our own products and their competitors, and the categories still pending. https://www.anchorterminal.com/benchmark/\n\n[2] Anchor Terminal's repository, counted on 9 November 2026, not published. Source for the numbers of listings (files under data/tools), design specs, lines of Go and the newest pull request number.\n\n[3] Anchor Terminal, research/RULES.md, 8 November 2026, not published. Source for what a research agent writes and doesn't touch, how a batch is landed, and the rules moving from a chain of batch files into one file, with the 36,000 tokens a research agent read before.\n\n[4] Anchor Terminal, how reviews work. Source for the panel, the audience reviewers and the arbiter, the models they run on, how the arbiter marks reviews, and the arbiter not covering every listing yet. https://www.anchorterminal.com/reviews/how-it-works\n\n[5] Anchor Terminal's README and go.mod, 9 November 2026, not published. Source for the workers and how often they run, the data model, the server rendering every page and its twins, the two outside packages, the checks run before a new site goes live, and the pull request rule being a convention on GitHub Free.\n\n[6] Anchor Terminal, CLAUDE.md, 9 November 2026, not published. Source for the rules for branches and pull requests, the test suite's time and memory, the size of a full build and the spelling lint being off.\n\n[7] Anchor Terminal, the Svalbard release plan, not published. Source for keeping editorial data in files under git and for computing everything a page says that can be computed.\n\n[8] Anchor Terminal, VOICE.md, the writing guide, 9 November 2026, not published. Source for what `go run ./gen -check` fails a page or a post on.\n\n[9] Anchor Terminal, docs/design/README.md, 9 November 2026, not published. Source for one spec per feature and what a spec records, with Vlad's words for decisions he took in conversation.\n\n[10] Anchor Terminal, the security review of the merges from 6 to 8 November 2026, not published. Source for the method of two independent reviewers and a third agent reproducing each finding.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-09",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.7",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "Blog",
        "url": "https://www.anchorterminal.com/blog/"
      },
      {
        "name": "Inside Anchor Terminal",
        "url": "https://www.anchorterminal.com/blog/inside/"
      },
      {
        "name": "A machine that builds the benchmark",
        "url": ""
      }
    ],
    "description": "Vlad Cealicu on how Anchor Terminal is put together so that agents research, grade, review and publish 1,651 listings, why it's laid out that way, and how two of us keep it sane.",
    "facts": [
      "Vlad Cealicu",
      "9 November 2026"
    ],
    "h1": "A machine that builds the benchmark",
    "image": "https://www.anchorterminal.com/assets/og/blog-a-machine-that-builds-the-benchmark.png",
    "path": "/blog/a-machine-that-builds-the-benchmark",
    "published": "2026-11-09",
    "section": "blog",
    "title": "How agents build our benchmark and two people keep it sane | Anchor Terminal",
    "toc": null,
    "updated": "2026-11-09",
    "url": "https://www.anchorterminal.com/blog/a-machine-that-builds-the-benchmark"
  },
  "tokens": {
    "markdown": 2700,
    "slim": 1030
  },
  "version": 1
}