Blog · From the data · 22 November 2026

An open model on a £3,879 box scored 98 on our test. Claude scored 100

The best open-weight model on our lab box missed two of 59 questions and three Claude models missed none. A 14.4 GB model came second, and every model on the box, down to a 4.5 GB one, got at least 17 of the 20 single lookups.

98the Lab score of Qwen3.8-Flash-Next on the lab box, with 57 of 59 questions right
92the Lab score of Gemma 4 26B A4B, from a 14.4 GB file
100the Lab score Claude Haiku 5.5, Sonnet 5.5 and Fable 5.2 each got on the same 59 questions, run by hand
£3,879a Minisforum MS-S1 Max with 128 GB of memory and a 4 TB disk
Anchor Terminal's lab evaluation, each model's latest score [1], the hosted runs [2] and Minisforum's UK store [3], read 21 November 2026.

The lab box next to our server is a Minisforum MS-S1 Max with 128 GB of unified memory, and the same machine was £3,879 on Minisforum's UK store on 21 November 2026 [3]. Eleven open-weight models have taken our evaluation on it, and the best of them, Qwen3.8-Flash-Next, got 57 of 59 questions right for a Lab score of 98 [1]. Gemma 4 26B A4B scored 92 from a file of 14.4 GB [1] [4].

Three Claude models took the same 59 questions by hand on 21 November 2026 and each scored 100 [2]. Two questions separate them from the best model on a £3,879 box [3], so I think a model you run yourself is now good enough for this kind of agent work.

Anchor Terminal's lab evaluation [1] and the hosted runs [2], read 21 November 2026Lab score on the same 59 questions, 0 to 100

Lab score

  1. Claude Fable 5.2, by hand100
  2. Claude Sonnet 5.5, by hand100
  3. Claude Haiku 5.5, by hand100
  4. Qwen3.8-Flash-Next98
  5. Gemma 4 26B A4B92
  6. Gemma 4 31B87
  7. Qwen3.6-35B-A3B77
  8. GLM-4.5-Air67
  9. gpt-oss-120b66
  10. Qwen3.5-9B66
  11. Nemotron 3 Super 120B A12B63
  12. Qwen3.5-4B62
  13. gpt-oss-20b61
  14. GLM-4.7-Flash55

Bars start at zero. Values are labelled in each panel.

The test is 59 questions an agent might ask Anchor Terminal, such as what grade we give a listing or which of two model APIs has the higher score [2]. The model has to look each answer up through our MCP server, our pages or our API [2]. Twenty of them take one lookup, 27 take a few or a comparison, and 12 take a filter, a join across our API's files or arithmetic over hundreds of listings [2]. How a question is asked and marked is in a post of its own, and this one is about what the scores mean.

If you're choosing a model to run yourself

ModelLab scoreRight, of 59Hardest ones, of 12File
Qwen3.8-Flash-Next98571275.8 GB
Gemma 4 26B A4B92551014.4 GB
Gemma 4 31B87521017.7 GB
Qwen3.6-35B-A3B7750536.9 GB
GLM-4.5-Air6744563.6 GB
gpt-oss-120b6643563.4 GB
Qwen3.5-9B664439.5 GB
Nemotron 3 Super 120B A12B6341579.0 GB
Qwen3.5-4B624144.5 GB
gpt-oss-20b6140412.1 GB
GLM-4.7-Flash5538331.8 GB

Scores and questions right are from the evaluation's page [1], and file sizes from the lab's model list [4].

If what your agent does is look one thing up through an MCP server, almost any of these will do it. Every model got between 17 and 20 of the 20 single lookups, and that includes Qwen3.5-4B, which is a 4.5 GB file [1] [4]. I wouldn't pay for a frontier model to do that job, and I wouldn't buy a 128 GB machine for it either.

The models split on work with several steps. Qwen3.8-Flash-Next got all 12 of the hardest questions, the two Gemmas got 10, and nobody else got more than 5 [1]. So for an agent that has to filter, join and add up over hundreds of rows, the choice today is between three models, and the second of them is a 14.4 GB file.

I wouldn't choose by the size of the file or the name on it. gpt-oss-120b at 63.4 GB scored 66 and Nemotron 3 Super at 79.0 GB scored 63, where a 14.4 GB Gemma scored 92 [1] [4]. I couldn't have guessed that order from the files, which is why I'd rather have a test I can run again the week a model comes out than a table I read once.

Anchor Terminal's lab evaluation [1], read 21 November 2026One question on the lab box, from the fastest model to the slowest
  1. gpt-oss-20b48 s

    Lab score 61

  2. gpt-oss-120b55 s

    Lab score 66

  3. Qwen3.6-35B-A3B56 s

    Lab score 77

  4. Qwen3.5-4B69 s

    Lab score 62

  5. Qwen3.8-Flash-Next71 s

    Lab score 98

  6. Gemma 4 26B A4B91 s

    Lab score 92

  7. GLM-4.7-Flash102 s

    Lab score 55

  8. Qwen3.5-9B2.0 min

    Lab score 66

  9. GLM-4.5-Air2.3 min

    Lab score 67

  10. Nemotron 3 Super 120B A12B4.8 min

    Lab score 63

  11. Gemma 4 31B5.3 min

    Lab score 87

Bar length is time on a log scale, and the bars grow at the same speed.

What you give up is time. The fastest of the eleven takes 48 seconds a question and the best takes 71 [1], where a Claude pass over all 59 took between two and a half and ten minutes [2]. The box holds one model in memory at a time, and the eleven evaluations took about 67 hours between them, going by the seconds a question on the page [1]. That's fine for work nobody is waiting on, which is what our lab does, and it isn't fine for anything a person watches.

What you get for it is that nothing leaves the machine. Our lab tests other people's MCP servers, the transcripts hold their tool descriptions and a model's words about them, and I didn't want those going to anybody's API, so the harness refuses a model that isn't on the box [2]. That rule could have cost us a lot in quality, and on this test it cost two questions.

If agents call your API

The model on the other end of your API won't always be a frontier one. A 14.4 GB file on somebody's own machine got 55 of our 59 questions right [1] [4], and I think a lot of agent traffic will come from models like it, because there's no bill for each call and the data stays where it is.

I don't think MCP is where this ends up. What an agent does with a service is write a small integration against its API to get what it needs out of it, and a model on our box can already work that way, with a Go program that reads our API [2]. So what matters is whether a model can get what it needs from your API with what you've written about it.

What decided the hard questions was how much the site let a model ask for in one go. Some questions in the harder set need one field from each of 60 or 80 or 177 listings [2]. A model that fetches the listings one at a time spends its calls on that. A model that notices our search filters on that field and says how many match gets the answer without fetching them [2]. So a filter and a count are worth more to a small model than another paragraph of description.

The other thing is to say no clearly. Four of our questions ask about something that doesn't exist, and the only way to find that out is to ask for it and be told so [2]. Our API answers with a 404 and our MCP server with a tool error [2], and both are fine, because a model can read either as a plain no and move on.

How Claude took it

Claude isn't a model the box can run, so it took the test by hand. Each pass was a new instance with no history, given what a model on the box is given and one logged way to reach the site, and it was graded by the same code [2]. A pass was one conversation for all 59 questions, where a model on the box starts each question fresh [2]. Fable 5.2 got 59 of 59 in all three passes, and Haiku 5.5 and Sonnet 5.5 scored 100 with their three passes counted together [2].

The harder set

A test where three models score 100 can't tell me when it's time to change model, so there's a second set of 66 questions, kept beside the normal one and never part of the Lab score [2].

ModelRight in each pass, of 66Right, passes togetherScoreRight in every pass
Claude Fable 5.266, 66, 666610066
Claude Sonnet 5.562, 64, 64659859
Claude Haiku 5.564, 62, 55639553

Three passes each by hand, from the harness's notes [2]. One question was reworded after these runs, and either reading of it is counted.

What cost Haiku and Sonnet their questions was a count over 60 listings that Haiku left blank in all three passes, rows counted that the question ruled out, a deduction left off a grade and a licence read from the wrong part of a file [2]. Most of that is planning a long job and reading what a question counts, and I'd expect that to be where the local models and the hosted ones come apart too.

What the harness does now

The scores above are from the first run of the eleven models. The harness has changed since, and every model runs again on it.

A model is told how many replies, calls and fetches a question gets, and it gets more of each, 32 replies and 200 calls and 200 fetches where it had 16 and 40 [5]. A reply can be 16,384 tokens, and a model's context is as long as the box has memory for, up to 131,072 tokens [5]. A program that fails to build only because of an unused import is fixed and built again without costing the model a reply [5]. The limits that stay are the ones that protect somebody else's server, because I want a model to do its best at testing the tools we list.

On the four questions about something that doesn't exist, being told so counts as the lookup, and the eleven saved runs are marked again from their own records, so no model runs twice for it [6]. When a model loses a question, the harness says how, whether it ran out of replies, gave no answer or answered with nothing looked up [5].

The lab is also learning to test plain HTTP APIs the way it tests MCP servers, with a model writing a small program against an API from what the API says about itself [7]. Over half of what we list is plain APIs [2], and that's the work I think agents will be doing.

No local model has taken the harder set yet. That run will say how far the local models are from the hosted ones.

References

Read on 21 and 22 November 2026.

  1. [1]Anchor Terminal, Lab score, open-weight models on our own evaluation. Source for each model's Lab score, questions right, right at each level and seconds a question. anchorterminal.com
  2. [2]Anchor Terminal, anchorlab, README.md and DESIGN.md. Source for what the evaluation asks and how it's scored, the model having to be on the box, the hosted models' runs by hand and their scores on both sets, and the hard set. github.com
  3. [3]Minisforum, MS-S1 Max, UK store. Source for £3,879 for the 128 GB model with a 4 TB disk. minisforum.uk
  4. [4]Anchor Terminal, anchorlab, MODELS.md. Source for each model's build and file size. github.com
  5. [5]Anchor Terminal, anchorlab, pull request 35. Source for a model being told its limits, the replies, calls, fetches, tokens and context it gets, the repaired build and the reasons a question was lost. github.com
  6. [6]Anchor Terminal, anchorlab, pull request 34. Source for being told a thing isn't there counting as the lookup, and the saved runs being marked again. github.com
  7. [7]Anchor Terminal, anchorlab, pull request 37. Source for the lab testing a plain HTTP API from its OpenAPI document. github.com

Your experience matters

Used a tool in the Terminal?

Tell people and their agents what worked and what didn't. Write a review of a tool you've used and confirm it by email or wallet.