{
  "data": {
    "author": "Vlad Cealicu",
    "faq": null,
    "kicker": "Methodology · open-weight models · plan of 21 October 2026",
    "lede": "The open-weight checklist leans on benchmark results other people publish until we have our own. This is how we'll get our own, on one machine an agent's operator could own, with the same harness for every model.",
    "readingMinutes": 1,
    "words": 0
  },
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/benchmark/open-weights-runs/",
    "json": "https://www.anchorterminal.com/benchmark/open-weights-runs/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/benchmark/open-weights-runs/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/benchmark/open-weights-runs/index.min.md"
  },
  "markdown": "This is a plan. Nothing on this page is a result yet, and no score on the site comes from these runs until they've happened and their outputs are published. The [open-weight checklist](/benchmark/#checklist-weights) says how the results will count.\n\n## The machine\n\nAn AMD Ryzen AI Max+ 395 (Strix Halo, Radeon 8060S, LLVM target gfx1151) with 128 GB of LPDDR5X unified memory, running Linux. We picked it because it's the kind of machine an agent's operator can buy and keep on a desk, and because a model that runs well here runs on a lot of hardware.\n\nThe GPU reaches system memory through GTT, which the kernel caps at about half of RAM by default. AMD's own guide raises the cap with `amd-ttm --set 100`, which writes a TTM page limit of 100 GiB, so 107.4 GB is the most a model and its context can use, and that's the ceiling below.\n\nThe runtimes are llama.cpp built for HIP (`-DGGML_HIP=ON -DGPU_TARGETS=gfx1151`) and for Vulkan, with the build tag recorded for every run, and vLLM on ROCm for safetensors at BF16. vLLM and SGLang don't document FP8, MXFP4 or W4A16 kernels on gfx1151, so we don't count on them until we've seen them work.\n\n## What fits\n\nFrom the file sizes on 21 October 2026, at the file size plus a fifth for the runtime and a working context. 22 of the 31 open-weight models we list fit within 107.4 GB at their smallest build, and 17 of those have a build the runtimes' documentation says runs on this chip.\n\n| Fits and runs | Precision we'll test | Size |\n| --- | --- | --- |\n| Qwen3.5-4B, Qwen3.5-9B, Qwen3-VL-8B-Instruct, SmolLM3-3B, MiniCPM5-2B, Ornith-1.5-9B | BF16 through vLLM, and a 4-bit GGUF through llama.cpp | 5 to 20 GB |\n| Gemma 4 26B A4B, Gemma 4 31B, Qwen3.8-27B, Qwen3.6-35B-A3B, GLM-4.7-Flash, Nemotron 3 Nano 30B A3B, Muse Glimmer 30B | BF16 through vLLM, and a 4-bit GGUF through llama.cpp | 52 to 72 GB at BF16 |\n| gpt-oss-20b, gpt-oss-120b | MXFP4, as published, through llama.cpp | 14 and 65 GB |\n| Nemotron 3 Super 120B A12B | Ollama's 4-bit build (Q4_K_M), the only one that fits | 87 GB |\n| Hy3 | a 1-bit GGUF that fits with almost nothing to spare, so speed only, and its listing says so | 89 GB |\n\nKolibri-1 (FP8 and BF16 only) and the four small Ming-3.0 speech models (inclusionAI's code is tested on CUDA) fit by size but wait until we've seen them load on this chip. Kimi K3, GLM-5.3, GLM-5.3-Flash, DeepSeek V3.2, V4 Flash 0731 and V4.1 Flash, MiniMax-M2.5, Qwen3.8-Flash-Next and Ming-3.0-audio-flash are too big for the machine at any precision their publishers or the runtimes ship, so they keep the independent results they have and their listings say so.\n\n## The suites\n\nEvery suite runs against the model's OpenAI-compatible endpoint on the same machine, through the same harness and settings for every model. The time per model assumes about 30 tokens a second out and 500 in, from published llama-bench results for this chip. Small and mixture-of-experts models will go faster, and the first model through each suite tells us how far off the estimate is.\n\n| Area | Suite | What it tells an agent | Time per model, estimated |\n| --- | --- | --- | --- |\n| Speed | llama-bench (llama.cpp), prompt 512 and 2,048, generation 128, at depths 0 to 32K | tokens per second and time to first token on this machine | 0.5 h |\n| Tool calling | BFCL V4, single, parallel, multiple, multi-turn and irrelevance categories (web search left out, it needs a paid search key) | whether the calls it makes are the calls you asked for | most of 33 h for the full set |\n| Agent tasks | τ²-bench 2.0, the 150 English text tasks (airline, retail, telecom) | finishing a multi-turn task with a user and tools | 41 h |\n| Agent tasks | Terminal-Bench 4.0 (August 2026), through Harbor | working in a shell to a verified end state | measured on the first model (2.0's 89 tasks were about 22 h) |\n| Instruction following | IFEval (541) and IFBench (300) | doing what it was told, checked by code | 3 h |\n| Reasoning and knowledge | GPQA Diamond (198), and MMLU-Pro on the first models to check quantisation against published BF16 results | how much it knows and reasons | 2 h, 56 h for MMLU-Pro |\n| Coding | Aider Polyglot (225), SWE-bench Verified's community 50-task set with mini-swe-agent, LiveCodeBench v6's 175 newest problems | editing real code until the tests pass | 5 h, 9 h, part of 60 h |\n| Long context | RULER at 50 samples a task up to 32K, and up to 128K for models that claim it | whether it uses a long context or only accepts one | 22 h |\n| Prompt injection | AgentDojo, all four suites, with and without attacks | whether text in a tool result can take it over | 12 h |\n| Over-refusal | XSTest (250 safe and 200 unsafe prompts) | whether it refuses things it shouldn't | 1 h |\n| Vision | MMMU validation (900), ChartQA (2,500), DocVQA validation | reading images, charts and documents | 4.5 h, 3 h, 12 h |\n| Speech to text | the Open ASR Leaderboard's sets (LibriSpeech, Common Voice, FLEURS) with its normaliser | word error rate and real-time factor | 3.5 h |\n| Text to speech | Seed-TTS-eval (English and Chinese) | intelligibility and speaker similarity | 3.5 h |\n\nTwo suites can't run on one machine the way their leaderboards run them. τ²-bench's user simulator and judges default to hosted models, so we run a fixed local simulator (the same one for every model, named in the results), which makes our numbers comparable with each other and not with τ²-bench's leaderboard. RULER at its default settings would take about 950 hours, so we run the reduced set above. Toolathlon and parts of MCP-Atlas need accounts with outside services, and SimpleQA's grader is a hosted model, so for those we keep using the results independent parties publish.\n\n## Settings\n\n- The publisher's recommended sampling settings and chat template, or the runtime's defaults where the publisher gives none, written down with every run.\n- Thinking on at the publisher's default effort, and off as a second run where the model can turn it off, because an agent often wants the fast answer.\n- 32K context unless a suite needs more.\n- Three runs of the short suites (speed, IFEval, IFBench, GPQA Diamond, XSTest) to see the spread, one of the long ones.\n- Every run records the model's file hashes, the build and precision, the runtime and its build tag, the harness version and the machine's settings, and the raw outputs are published with the results.\n\n## The order\n\n1. Speed for every model that runs, so Reach can take it on the first day.\n2. Instruction following, GPQA Diamond and over-refusal for every chat model, the cheapest suites that say the most.\n3. Tool calling and prompt injection for every chat model, because that's what an agent needs most from a cousin.\n4. Agent tasks and coding for the models that do well in the first three, starting with the ones that fit in 32 GB, because more operators can run those.\n5. Long context, vision and speech for the models that claim them.\n\nAt these estimates the first three steps take about a month of machine time for the 17 models, mostly BFCL and AgentDojo, and less for the small and mixture-of-experts models, which run several times faster than 30 tokens a second. The rest runs for weeks after. We'll publish each step as it finishes rather than wait for all of it, and if a suite turns out too slow to run on every model we'll say which subset we used.\n\n## How the results count\n\n- Our results count in full in Capability, in the area each suite belongs to, scored the same way as independent results, as the model's share of the way from the suite's floor to the best result we or an independent party have recorded.\n- Speed joins Reach as its own item.\n- Prompt-injection success rate and over-refusal join Risk.\n- A model we ran at 4-bit is marked so, and where an independent BF16 result exists for the same suite we show both, so the cost of quantising is visible.\n- A model too big for the machine keeps the independent results it has, and its listing says we couldn't run it.\n\nWe checked each suite's version, licence and what it needs to run locally on 21 October 2026, and we'll pin the versions on the day the runs start.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-10-21",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.6",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "Open-weight runs",
        "url": ""
      }
    ],
    "description": "The plan for testing open-weight models on our own machine, an AMD Ryzen AI Max+ 395 with 128 GB of unified memory. Which models fit and at what precision, the suites we run (speed, tool calling, agent tasks, instruction following, coding, prompt injection, over-refusal, speech and vision), the settings, the order, and how the results feed the open-weight checklist.",
    "facts": [
      "AMD Ryzen AI Max+ 395",
      "128 GB unified memory",
      "plan, not results yet"
    ],
    "h1": "Running open-weight models ourselves",
    "image": "https://www.anchorterminal.com/assets/og/benchmark-open-weights-runs.png",
    "path": "/benchmark/open-weights-runs/",
    "published": "2026-10-01",
    "section": "about",
    "title": "How we'll benchmark open-weight models on a 128 GB Strix Halo machine | Anchor Terminal",
    "toc": [
      {
        "id": "machine",
        "level": "h2",
        "text": "The machine"
      },
      {
        "id": "fits",
        "level": "h2",
        "text": "What fits"
      },
      {
        "id": "suites",
        "level": "h2",
        "text": "The suites"
      },
      {
        "id": "settings",
        "level": "h2",
        "text": "Settings"
      },
      {
        "id": "order",
        "level": "h2",
        "text": "The order"
      },
      {
        "id": "scoring",
        "level": "h2",
        "text": "How the results count"
      }
    ],
    "updated": "2026-10-21",
    "url": "https://www.anchorterminal.com/benchmark/open-weights-runs/"
  },
  "tokens": {
    "markdown": 2200,
    "slim": 1630
  },
  "version": 1
}