Lab

Lab score: open-weight models on our own evaluation

Each open-weight model the lab runs answers 59 questions about Anchor Terminal through our MCP server, pages and API, three times over, calling the tools itself or writing programs that call them, as it chooses. The right answers come from the site when each question is asked, so they follow the listings, and a model can't learn them. The Lab score, 0 to 100, is the questions it got right with harder ones counting more.

No model has a Lab score yet. The first results appear here, dated, as soon as the lab box reports them.

Every evaluation

A run in code mode, where the model may only write programs, is kept here beside the Lab score's runs and scores nothing.

DateModelModeScoreRightPassesA question takesMinutes in the model a pass
18 November 2026Qwen3.8-27Bcode only8954 of 5933.1 min175.5

How it works

  • The questions are public, in the lab's repository: eval/tasks.go. Level 1 is one lookup, level 2 a few lookups or a comparison, and level 3 a filter, a join across our API's files or arithmetic over hundreds of listings. A few have answers that don't move, and a few are traps, such as a listing that doesn't exist or a write tool to leave alone.
  • The model gets our MCP server's tools and one of the lab's own, a fetch that reaches only this site, as tools it can call itself, and one more that runs a Go program it writes. The program runs with no network and can call the same tools. For each question the model chooses: its own calls, a program, or both. Every call goes through the same gateway either way, with the same limits.
  • An answer is checked against the site as it was when the question was asked and as it is when the model answered, so a listing re-graded in between doesn't cost the model the question. A question whose subject left the site, or that has two right answers at the moment, is skipped, not failed. An answer with no lookup behind it counts as wrong, even when it's right.
  • Time is recorded beside the score and never in it: how long a question takes, from the question to the graded answer, and the model's tokens a second. A model that answers fast and wrong isn't helped by its speed, and one that's slow and right isn't marked down for it. Both numbers are here to weigh.
  • A question counts as right when the model got it right in more than half of the three passes. The Lab score counts a level 2 question twice and a level 3 question three times, so models that all get the easy ones right are told apart by the hard ones.
  • The models run on the lab box, an AMD Ryzen AI Max+ 395 with 128 GB, through llama.cpp, with each publisher's own sampling settings. The models and their builds are pinned by file hash.
  • A model's Lab score is on its listing, and the local model calculator ranks by it where a model has one, ahead of the Capability score read from other people's leaderboards.

Set anchor-lab-eval/1. JSON: /api/v1/lab-evals.json.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.