Lab
Lab score: open-weight models on our own evaluation
Each open-weight model the lab runs answers 59 questions about Anchor Terminal through our MCP server, pages and API, three times over, calling the tools itself or writing programs that call them, as it chooses. The right answers come from the site when each question is asked, so they follow the listings, and a model can't learn them. The Lab score, 0 to 100, is the questions it got right with harder ones counting more.
The harness on GitHub The 59 questions The models and their builds The results as JSON
- One lookup, a point each
- A few lookups or a comparison, two points each
- Hard (a filter, a join or arithmetic over hundreds of listings), three points each
Under each bar, how many questions of each kind the model got right, how long a question takes it on the lab box, how fast it writes, and whether it answered with its own tool calls, with programs or with both. Time is beside the score and never in it.
The setup
One machine, an AMD Ryzen AI Max+ 395 with 128 GB. Every model runs on it through llama.cpp with its publisher's own sampling settings, from a build pinned by file hash.
The model gets our MCP server's tools and a fetch that reaches this site and nothing else. It can call them itself, or write a Go program that calls them, and the program runs with no network.
59 of them about this site, each with one right answer. The answers are read from the site when the question is asked, so they follow the listings and a model can't have learnt them.
A question is right when the model got it right in more than half of 3 passes. A harder question counts more, and an answer with no lookup behind it counts as wrong.
The harness is open source, in Go with the standard library and nothing else, under the MIT licence. More about the box, and how what runs on it is kept apart, is on the lab's page.
By time
How long a question takes on average on the lab box, from reading the question to the graded answer, and how fast the model writes. Fastest first.
| Model | A question takes | Tokens a second | Lab score |
|---|---|---|---|
| gpt-oss-20b | 48 s | 45.0 | 61 |
| Qwen3-Coder-Next | 53 s | 32.7 | 61 |
| gpt-oss-120b | 55 s | 29.9 | 66 |
| Qwen3.6-35B-A3B | 56 s | 33.8 | 77 |
| Qwen3-Coder-30B-A3B-Instruct | 64 s | 34.9 | 59 |
| Qwen3-30B-A3B-Instruct-2507 | 68 s | 31.7 | 47 |
| Qwen3.5-4B | 69 s | 26.3 | 62 |
| Qwen3.8-Flash-Next | 71 s | 18.6 | 98 |
| Gemma 4 26B A4B IT | 91 s | 46.5 | 92 |
| GLM-4.7-Flash | 102 s | 14.4 | 55 |
| Qwen3.5-9B | 2.0 min | 17.2 | 66 |
| GLM-4.5-Air | 2.3 min | 9.4 | 67 |
| Nemotron 3 Nano 30B A3B | 3.2 min | 48.5 | 59 |
| Nemotron 3 Super 120B A12B | 4.8 min | 15.0 | 63 |
| Gemma 4 31B IT | 5.3 min | 9.1 | 87 |
Every evaluation
A run in code mode, where the model may only write programs, is kept here beside the Lab score's runs and scores nothing.
| Date | Model | Mode | Score | Right | Passes | A question takes | Minutes in the model a pass |
|---|---|---|---|---|---|---|---|
| 22 November 2026 | Qwen3-Coder-30B-A3B-Instruct | either | 59 | 38 of 59 | 3 | 64 s | 60.0 |
| 22 November 2026 | Qwen3-30B-A3B-Instruct-2507 | either | 47 | 31 of 59 | 3 | 68 s | 59.8 |
| 22 November 2026 | Qwen3-Coder-Next | either | 61 | 41 of 59 | 3 | 53 s | 46.1 |
| 22 November 2026 | Nemotron 3 Nano 30B A3B | either | 59 | 40 of 59 | 3 | 3.2 min | 183.4 |
| 21 November 2026 | GLM-4.5-Air | either | 67 | 44 of 59 | 3 | 2.3 min | 132.1 |
| 21 November 2026 | gpt-oss-20b | either | 61 | 40 of 59 | 3 | 48 s | 43.0 |
| 21 November 2026 | Qwen3.5-4B | either | 62 | 41 of 59 | 3 | 69 s | 64.7 |
| 21 November 2026 | Qwen3.5-9B | either | 66 | 44 of 59 | 3 | 2.0 min | 115.0 |
| 21 November 2026 | Nemotron 3 Super 120B A12B | either | 63 | 41 of 59 | 3 | 4.8 min | 275.0 |
| 20 November 2026 | Qwen3.8-Flash-Next | either | 98 | 57 of 59 | 3 | 71 s | 63.5 |
| 20 November 2026 | gpt-oss-120b | either | 66 | 43 of 59 | 3 | 55 s | 51.1 |
| 20 November 2026 | Qwen3.6-35B-A3B | either | 77 | 50 of 59 | 3 | 56 s | 51.3 |
| 20 November 2026 | GLM-4.7-Flash | either | 55 | 38 of 59 | 3 | 102 s | 93.4 |
| 19 November 2026 | Gemma 4 26B A4B IT | either | 92 | 55 of 59 | 3 | 91 s | 84.2 |
| 19 November 2026 | Gemma 4 31B IT | either | 87 | 52 of 59 | 3 | 5.3 min | 307.7 |
| 18 November 2026 | Qwen3.8-27B | code only | 89 | 54 of 59 | 3 | 3.1 min | 175.5 |
How it works
- The questions are public, in the lab's repository: eval/tasks.go. Level 1 is one lookup, level 2 a few lookups or a comparison, and level 3 a filter, a join across our API's files or arithmetic over hundreds of listings. A few have answers that don't move, and a few are traps, such as a listing that doesn't exist or a write tool to leave alone.
- The model gets our MCP server's tools and one of the lab's own, a fetch that reaches only this site, as tools it can call itself, and one more that runs a Go program it writes. The program runs with no network and can call the same tools. For each question the model chooses: its own calls, a program, or both. Every call goes through the same gateway either way, with the same limits.
- An answer is checked against the site as it was when the question was asked and as it is when the model answered, so a listing re-graded in between doesn't cost the model the question. A question whose subject left the site, or that has two right answers at the moment, is skipped, not failed. An answer with no lookup behind it counts as wrong, even when it's right.
- Time is recorded beside the score and never in it: how long a question takes, from the question to the graded answer, and the model's tokens a second. A model that answers fast and wrong isn't helped by its speed, and one that's slow and right isn't marked down for it. Both numbers are here to weigh.
- A question counts as right when the model got it right in more than half of the three passes. The Lab score counts a level 2 question twice and a level 3 question three times, so models that all get the easy ones right are told apart by the hard ones.
- The models run on the lab box, an AMD Ryzen AI Max+ 395 with 128 GB, through llama.cpp, with each publisher's own sampling settings. The models and their builds are pinned by file hash.
- A model's Lab score is on its listing, and the local model calculator ranks by it where a model has one, ahead of the Capability score read from other people's leaderboards.
Set anchor-lab-eval/1. JSON: /api/v1/lab-evals.json.