# An open model on a £3,879 box scored 98 on our test. Claude scored 100

> Vlad Cealicu on what eleven open-weight models scored on Anchor Terminal's own agent test, run on a Minisforum MS-S1 Max with 128 GB of memory. The best scored 98, a 14.4 GB model scored 92, and Claude Haiku, Sonnet and Fable each scored 100 by hand. What that means for choosing a model to run yourself and for anyone whose API an agent calls, and what the lab's harness does now.

- Canonical: https://www.anchorterminal.com/blog/local-models-on-one-box-against-claude
- Markdown: https://www.anchorterminal.com/blog/local-models-on-one-box-against-claude.md (~3,400 tokens)
- Slim: https://www.anchorterminal.com/blog/local-models-on-one-box-against-claude.min.md (~1,680 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/blog/local-models-on-one-box-against-claude.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-22

By Vlad Cealicu*, 22 November 2026. 1,723 words, about 7 min to read. * With the editing help of Claude Opus or Claude Fable.

- 98: the Lab score of Qwen3.8-Flash-Next on the lab box, with 57 of 59 questions right
- 92: the Lab score of Gemma 4 26B A4B, from a 14.4 GB file
- 100: the Lab score Claude Haiku 5.5, Sonnet 5.5 and Fable 5.2 each got on the same 59 questions, run by hand
- £3,879: a Minisforum MS-S1 Max with 128 GB of memory and a 4 TB disk

Anchor Terminal's lab evaluation, each model's latest score [1], the hosted runs [2] and Minisforum's UK store [3], read 21 November 2026.

The lab box next to our server is a Minisforum MS-S1 Max with 128 GB of unified memory, and the same machine was £3,879 on Minisforum's UK store on 21 November 2026 [[3]](#ref-3). Eleven open-weight models have taken our evaluation on it, and the best of them, Qwen3.8-Flash-Next, got 57 of 59 questions right for a Lab score of 98 [[1]](#ref-1). Gemma 4 26B A4B scored 92 from a file of 14.4 GB [[1]](#ref-1) [[4]](#ref-4).

Three Claude models took the same 59 questions by hand on 21 November 2026 and each scored 100 [[2]](#ref-2). Two questions separate them from the best model on a £3,879 box [[3]](#ref-3), so I think a model you run yourself is now good enough for this kind of agent work.

Lab score on the same 59 questions, 0 to 100

| Model | Lab score |
| --- | --- |
| Claude Fable 5.2, by hand | 100 |
| Claude Sonnet 5.5, by hand | 100 |
| Claude Haiku 5.5, by hand | 100 |
| Qwen3.8-Flash-Next | 98 |
| Gemma 4 26B A4B | 92 |
| Gemma 4 31B | 87 |
| Qwen3.6-35B-A3B | 77 |
| GLM-4.5-Air | 67 |
| gpt-oss-120b | 66 |
| Qwen3.5-9B | 66 |
| Nemotron 3 Super 120B A12B | 63 |
| Qwen3.5-4B | 62 |
| gpt-oss-20b | 61 |
| GLM-4.7-Flash | 55 |

Each series has a zero-based scale.

The test is 59 questions an agent might ask Anchor Terminal, such as what grade we give a listing or which of two model APIs has the higher score [[2]](#ref-2). The model has to look each answer up through our MCP server, our pages or our API [[2]](#ref-2). Twenty of them take one lookup, 27 take a few or a comparison, and 12 take a filter, a join across our API's files or arithmetic over hundreds of listings [[2]](#ref-2). How a question is asked and marked is in [a post of its own](/blog/we-test-a-model-on-our-own-site-first), and this one is about what the scores mean.

## If you're choosing a model to run yourself

| Model | Lab score | Right, of 59 | Hardest ones, of 12 | File |
| --- | --- | --- | --- | --- |
| Qwen3.8-Flash-Next | 98 | 57 | 12 | 75.8 GB |
| Gemma 4 26B A4B | 92 | 55 | 10 | 14.4 GB |
| Gemma 4 31B | 87 | 52 | 10 | 17.7 GB |
| Qwen3.6-35B-A3B | 77 | 50 | 5 | 36.9 GB |
| GLM-4.5-Air | 67 | 44 | 5 | 63.6 GB |
| gpt-oss-120b | 66 | 43 | 5 | 63.4 GB |
| Qwen3.5-9B | 66 | 44 | 3 | 9.5 GB |
| Nemotron 3 Super 120B A12B | 63 | 41 | 5 | 79.0 GB |
| Qwen3.5-4B | 62 | 41 | 4 | 4.5 GB |
| gpt-oss-20b | 61 | 40 | 4 | 12.1 GB |
| GLM-4.7-Flash | 55 | 38 | 3 | 31.8 GB |

Scores and questions right are from the evaluation's page [[1]](#ref-1), and file sizes from the lab's model list [[4]](#ref-4).

If what your agent does is look one thing up through an MCP server, almost any of these will do it. Every model got between 17 and 20 of the 20 single lookups, and that includes Qwen3.5-4B, which is a 4.5 GB file [[1]](#ref-1) [[4]](#ref-4). I wouldn't pay for a frontier model to do that job, and I wouldn't buy a 128 GB machine for it either.

The models split on work with several steps. Qwen3.8-Flash-Next got all 12 of the hardest questions, the two Gemmas got 10, and nobody else got more than 5 [[1]](#ref-1). So for an agent that has to filter, join and add up over hundreds of rows, the choice today is between three models, and the second of them is a 14.4 GB file.

I wouldn't choose by the size of the file or the name on it. gpt-oss-120b at 63.4 GB scored 66 and Nemotron 3 Super at 79.0 GB scored 63, where a 14.4 GB Gemma scored 92 [[1]](#ref-1) [[4]](#ref-4). I couldn't have guessed that order from the files, which is why I'd rather have a test I can run again the week a model comes out than a table I read once.

One question on the lab box, from the fastest model to the slowest

| Who | Time | Outcome |
| --- | --- | --- |
| gpt-oss-20b | 48 s | Lab score 61 |
| gpt-oss-120b | 55 s | Lab score 66 |
| Qwen3.6-35B-A3B | 56 s | Lab score 77 |
| Qwen3.5-4B | 69 s | Lab score 62 |
| Qwen3.8-Flash-Next | 71 s | Lab score 98 |
| Gemma 4 26B A4B | 91 s | Lab score 92 |
| GLM-4.7-Flash | 102 s | Lab score 55 |
| Qwen3.5-9B | 2.0 min | Lab score 66 |
| GLM-4.5-Air | 2.3 min | Lab score 67 |
| Nemotron 3 Super 120B A12B | 4.8 min | Lab score 63 |
| Gemma 4 31B | 5.3 min | Lab score 87 |

Time on a log scale.

What you give up is time. The fastest of the eleven takes 48 seconds a question and the best takes 71 [[1]](#ref-1), where a Claude pass over all 59 took between two and a half and ten minutes [[2]](#ref-2). The box holds one model in memory at a time, and the eleven evaluations took about 67 hours between them, going by the seconds a question on the page [[1]](#ref-1). That's fine for work nobody is waiting on, which is what our lab does, and it isn't fine for anything a person watches.

What you get for it is that nothing leaves the machine. Our lab tests other people's MCP servers, the transcripts hold their tool descriptions and a model's words about them, and I didn't want those going to anybody's API, so the harness refuses a model that isn't on the box [[2]](#ref-2). That rule could have cost us a lot in quality, and on this test it cost two questions.

## If agents call your API

The model on the other end of your API won't always be a frontier one. A 14.4 GB file on somebody's own machine got 55 of our 59 questions right [[1]](#ref-1) [[4]](#ref-4), and I think a lot of agent traffic will come from models like it, because there's no bill for each call and the data stays where it is.

I don't think MCP is where this ends up. What an agent does with a service is write a small integration against its API to get what it needs out of it, and a model on our box can already work that way, with a Go program that reads our API [[2]](#ref-2). So what matters is whether a model can get what it needs from your API with what you've written about it.

What decided the hard questions was how much the site let a model ask for in one go. Some questions in the harder set need one field from each of 60 or 80 or 177 listings [[2]](#ref-2). A model that fetches the listings one at a time spends its calls on that. A model that notices our search filters on that field and says how many match gets the answer without fetching them [[2]](#ref-2). So a filter and a count are worth more to a small model than another paragraph of description.

The other thing is to say no clearly. Four of our questions ask about something that doesn't exist, and the only way to find that out is to ask for it and be told so [[2]](#ref-2). Our API answers with a 404 and our MCP server with a tool error [[2]](#ref-2), and both are fine, because a model can read either as a plain no and move on.

## How Claude took it

Claude isn't a model the box can run, so it took the test by hand. Each pass was a new instance with no history, given what a model on the box is given and one logged way to reach the site, and it was graded by the same code [[2]](#ref-2). A pass was one conversation for all 59 questions, where a model on the box starts each question fresh [[2]](#ref-2). Fable 5.2 got 59 of 59 in all three passes, and Haiku 5.5 and Sonnet 5.5 scored 100 with their three passes counted together [[2]](#ref-2).

## The harder set

A test where three models score 100 can't tell me when it's time to change model, so there's a second set of 66 questions, kept beside the normal one and never part of the Lab score [[2]](#ref-2).

| Model | Right in each pass, of 66 | Right, passes together | Score | Right in every pass |
| --- | --- | --- | --- | --- |
| Claude Fable 5.2 | 66, 66, 66 | 66 | 100 | 66 |
| Claude Sonnet 5.5 | 62, 64, 64 | 65 | 98 | 59 |
| Claude Haiku 5.5 | 64, 62, 55 | 63 | 95 | 53 |

Three passes each by hand, from the harness's notes [[2]](#ref-2). One question was reworded after these runs, and either reading of it is counted.

What cost Haiku and Sonnet their questions was a count over 60 listings that Haiku left blank in all three passes, rows counted that the question ruled out, a deduction left off a grade and a licence read from the wrong part of a file [[2]](#ref-2). Most of that is planning a long job and reading what a question counts, and I'd expect that to be where the local models and the hosted ones come apart too.

## What the harness does now

The scores above are from the first run of the eleven models. The harness has changed since, and every model runs again on it.

A model is told how many replies, calls and fetches a question gets, and it gets more of each, 32 replies and 200 calls and 200 fetches where it had 16 and 40 [[5]](#ref-5). A reply can be 16,384 tokens, and a model's context is as long as the box has memory for, up to 131,072 tokens [[5]](#ref-5). A program that fails to build only because of an unused import is fixed and built again without costing the model a reply [[5]](#ref-5). The limits that stay are the ones that protect somebody else's server, because I want a model to do its best at testing the tools we list.

On the four questions about something that doesn't exist, being told so counts as the lookup, and the eleven saved runs are marked again from their own records, so no model runs twice for it [[6]](#ref-6). When a model loses a question, the harness says how, whether it ran out of replies, gave no answer or answered with nothing looked up [[5]](#ref-5).

The lab is also learning to test plain HTTP APIs the way it tests MCP servers, with a model writing a small program against an API from what the API says about itself [[7]](#ref-7). Over half of what we list is plain APIs [[2]](#ref-2), and that's the work I think agents will be doing.

No local model has taken the harder set yet. That run will say how far the local models are from the hosted ones.

---

More in [From the data](https://www.anchorterminal.com/blog/from-the-data/index.md) · Previous from the data [What to build that lasts. The jobs agents ask for that nobody finishes](https://www.anchorterminal.com/blog/the-jobs-nobody-finishes.md)

- Older: [We test a model on our own site before it tests anyone else's tools](https://www.anchorterminal.com/blog/we-test-a-model-on-our-own-site-first.md)
- Newer: [A small file took an open model from 35% to 51% on hard coding tasks. What a LoRA is, and what it changes outside code](https://www.anchorterminal.com/blog/what-a-lora-is-and-what-it-changes.md)
- All posts: https://www.anchorterminal.com/blog/index.md

Anchor Terminal grades the tools AI agents run on, with a source for every score. [The top tools](https://www.anchorterminal.com/top/index.md) · [Categories](https://www.anchorterminal.com/categories/index.md) · [Compare two tools](https://www.anchorterminal.com/compare/index.md) · [Prices](https://www.anchorterminal.com/prices/index.md) · [Sunsets](https://www.anchorterminal.com/sunsets/index.md) · [API, MCP server and CLI](https://www.anchorterminal.com/agents/index.md)

## References

Read on 21 and 22 November 2026.

[1] Anchor Terminal, Lab score, open-weight models on our own evaluation. Source for each model's Lab score, questions right, right at each level and seconds a question. https://www.anchorterminal.com/benchmark/lab/evaluation/models/

[2] Anchor Terminal, anchorlab, README.md and DESIGN.md. Source for what the evaluation asks and how it's scored, the model having to be on the box, the hosted models' runs by hand and their scores on both sets, and the hard set. https://github.com/Anchor-Terminal/anchorlab

[3] Minisforum, MS-S1 Max, UK store. Source for £3,879 for the 128 GB model with a 4 TB disk. https://www.minisforum.uk/products/minisforum-ms-s1-max

[4] Anchor Terminal, anchorlab, MODELS.md. Source for each model's build and file size. https://github.com/Anchor-Terminal/anchorlab/blob/main/MODELS.md

[5] Anchor Terminal, anchorlab, pull request 35. Source for a model being told its limits, the replies, calls, fetches, tokens and context it gets, the repaired build and the reasons a question was lost. https://github.com/Anchor-Terminal/anchorlab/pull/35

[6] Anchor Terminal, anchorlab, pull request 34. Source for being told a thing isn't there counting as the lookup, and the saved runs being marked again. https://github.com/Anchor-Terminal/anchorlab/pull/34

[7] Anchor Terminal, anchorlab, pull request 37. Source for the lab testing a plain HTTP API from its OpenAPI document. https://github.com/Anchor-Terminal/anchorlab/pull/37

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
