Lab

Lab score: open-weight models on our own evaluation

Each open-weight model the lab runs answers 59 questions about Anchor Terminal through our MCP server, pages and API, three times over, calling the tools itself or writing programs that call them, as it chooses. The right answers come from the site when each question is asked, so they follow the listings, and a model can't learn them. The Lab score, 0 to 100, is the questions it got right with harder ones counting more.

15 models · each one's latest run · newest 22 November 2026Lab score, 0 to 100, and which questions the points came from
  1. 18 of 2027 of 2712 of 1271 s a question18.6 tokens a second20 Novemberown calls 56%, both 44%

  2. 19 of 2026 of 2710 of 1291 s a question46.5 tokens a second19 Novemberown calls 57%, programs 1%, both 43%

  3. 18 of 2024 of 2710 of 125.3 min a question9.1 tokens a second19 Novemberown calls 56%, both 44%

  4. 20 of 2025 of 275 of 1256 s a question33.8 tokens a second20 Novemberown calls 75%, both 25%

  5. 19 of 2020 of 275 of 122.3 min a question9.4 tokens a second21 Novemberown calls 82%, both 18%

  6. 18 of 2020 of 275 of 1255 s a question29.9 tokens a second20 Novemberown calls 71%, both 29%

  7. 18 of 2023 of 273 of 122.0 min a question17.2 tokens a second21 Novemberown calls 72%, both 28%

  8. 18 of 2018 of 275 of 124.8 min a question15.0 tokens a second21 Novemberown calls 60%, both 40%

  9. 18 of 2019 of 274 of 1269 s a question26.3 tokens a second21 Novemberown calls 83%, both 17%

  10. 17 of 2019 of 274 of 1248 s a question45.0 tokens a second21 Novemberown calls 69%, both 31%

  11. 18 of 2020 of 273 of 1253 s a question32.7 tokens a second22 Novemberown calls 54%, both 46%

  12. 14 of 2021 of 273 of 1264 s a question34.9 tokens a second22 Novemberown calls 66%, both 34%

  13. 18 of 2019 of 273 of 123.2 min a question48.5 tokens a second22 Novemberown calls 70%, both 30%

  14. 19 of 2016 of 273 of 12102 s a question14.4 tokens a second20 Novemberown calls 75%, programs 1%, both 25%

  15. 14 of 2013 of 274 of 1268 s a question31.7 tokens a second22 Novemberown calls 49%, both 51%

  • One lookup, a point each
  • A few lookups or a comparison, two points each
  • Hard (a filter, a join or arithmetic over hundreds of listings), three points each

Under each bar, how many questions of each kind the model got right, how long a question takes it on the lab box, how fast it writes, and whether it answered with its own tool calls, with programs or with both. Time is beside the score and never in it.

The setup

The box

One machine, an AMD Ryzen AI Max+ 395 with 128 GB. Every model runs on it through llama.cpp with its publisher's own sampling settings, from a build pinned by file hash.

The cell

The model gets our MCP server's tools and a fetch that reaches this site and nothing else. It can call them itself, or write a Go program that calls them, and the program runs with no network.

The questions

59 of them about this site, each with one right answer. The answers are read from the site when the question is asked, so they follow the listings and a model can't have learnt them.

The score

A question is right when the model got it right in more than half of 3 passes. A harder question counts more, and an answer with no lookup behind it counts as wrong.

The harness is open source, in Go with the standard library and nothing else, under the MIT licence. More about the box, and how what runs on it is kept apart, is on the lab's page.

By time

How long a question takes on average on the lab box, from reading the question to the graded answer, and how fast the model writes. Fastest first.

ModelA question takesTokens a secondLab score
gpt-oss-20b48 s45.061
Qwen3-Coder-Next53 s32.761
gpt-oss-120b55 s29.966
Qwen3.6-35B-A3B56 s33.877
Qwen3-Coder-30B-A3B-Instruct64 s34.959
Qwen3-30B-A3B-Instruct-250768 s31.747
Qwen3.5-4B69 s26.362
Qwen3.8-Flash-Next71 s18.698
Gemma 4 26B A4B IT91 s46.592
GLM-4.7-Flash102 s14.455
Qwen3.5-9B2.0 min17.266
GLM-4.5-Air2.3 min9.467
Nemotron 3 Nano 30B A3B3.2 min48.559
Nemotron 3 Super 120B A12B4.8 min15.063
Gemma 4 31B IT5.3 min9.187

Every evaluation

A run in code mode, where the model may only write programs, is kept here beside the Lab score's runs and scores nothing.

DateModelModeScoreRightPassesA question takesMinutes in the model a pass
22 November 2026Qwen3-Coder-30B-A3B-Instructeither5938 of 59364 s60.0
22 November 2026Qwen3-30B-A3B-Instruct-2507either4731 of 59368 s59.8
22 November 2026Qwen3-Coder-Nexteither6141 of 59353 s46.1
22 November 2026Nemotron 3 Nano 30B A3Beither5940 of 5933.2 min183.4
21 November 2026GLM-4.5-Aireither6744 of 5932.3 min132.1
21 November 2026gpt-oss-20beither6140 of 59348 s43.0
21 November 2026Qwen3.5-4Beither6241 of 59369 s64.7
21 November 2026Qwen3.5-9Beither6644 of 5932.0 min115.0
21 November 2026Nemotron 3 Super 120B A12Beither6341 of 5934.8 min275.0
20 November 2026Qwen3.8-Flash-Nexteither9857 of 59371 s63.5
20 November 2026gpt-oss-120beither6643 of 59355 s51.1
20 November 2026Qwen3.6-35B-A3Beither7750 of 59356 s51.3
20 November 2026GLM-4.7-Flasheither5538 of 593102 s93.4
19 November 2026Gemma 4 26B A4B ITeither9255 of 59391 s84.2
19 November 2026Gemma 4 31B ITeither8752 of 5935.3 min307.7
18 November 2026Qwen3.8-27Bcode only8954 of 5933.1 min175.5

How it works

  • The questions are public, in the lab's repository: eval/tasks.go. Level 1 is one lookup, level 2 a few lookups or a comparison, and level 3 a filter, a join across our API's files or arithmetic over hundreds of listings. A few have answers that don't move, and a few are traps, such as a listing that doesn't exist or a write tool to leave alone.
  • The model gets our MCP server's tools and one of the lab's own, a fetch that reaches only this site, as tools it can call itself, and one more that runs a Go program it writes. The program runs with no network and can call the same tools. For each question the model chooses: its own calls, a program, or both. Every call goes through the same gateway either way, with the same limits.
  • An answer is checked against the site as it was when the question was asked and as it is when the model answered, so a listing re-graded in between doesn't cost the model the question. A question whose subject left the site, or that has two right answers at the moment, is skipped, not failed. An answer with no lookup behind it counts as wrong, even when it's right.
  • Time is recorded beside the score and never in it: how long a question takes, from the question to the graded answer, and the model's tokens a second. A model that answers fast and wrong isn't helped by its speed, and one that's slow and right isn't marked down for it. Both numbers are here to weigh.
  • A question counts as right when the model got it right in more than half of the three passes. The Lab score counts a level 2 question twice and a level 3 question three times, so models that all get the easy ones right are told apart by the hard ones.
  • The models run on the lab box, an AMD Ryzen AI Max+ 395 with 128 GB, through llama.cpp, with each publisher's own sampling settings. The models and their builds are pinned by file hash.
  • A model's Lab score is on its listing, and the local model calculator ranks by it where a model has one, ahead of the Capability score read from other people's leaderboards.

Set anchor-lab-eval/1. JSON: /api/v1/lab-evals.json.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.