{
  "data": {
    "latest": [],
    "method": "Questions with one right answer about Anchor Terminal, answered through its MCP server, pages and API, three passes each, the model calling the tools itself or writing programs that call them, as it chooses (mode either). The Lab score is the share right with a level 2 question counting twice and a level 3 question three times, from either-mode runs only; a code-mode run (programs only) is listed and scores nothing.",
    "page": "https://www.anchorterminal.com/lab/evaluation/",
    "questions": "https://github.com/Anchor-Terminal/anchorlab/blob/main/eval/tasks.go",
    "runs": [
      {
        "byLevel": [
          20,
          24,
          10
        ],
        "correct": 54,
        "ended": "2026-11-18T15:10:45Z",
        "forbidden": 0,
        "harness": "2026.11.18-0557",
        "jsonUrl": "https://www.anchorterminal.com/api/v1/tools/qwen3-8-27b.json",
        "mode": "code",
        "model": "qwen3.8-27b",
        "modelMinutes": 175.5,
        "name": "Qwen3.8-27B",
        "ofLevel": [
          20,
          27,
          12
        ],
        "outputTps": 7.1,
        "passes": 3,
        "pct": 91.5,
        "points": 89,
        "scored": 59,
        "secPerTask": 187.523,
        "slug": "qwen3-8-27b",
        "steady": 45,
        "tokens": 1924154
      }
    ],
    "set": "anchor-lab-eval/1",
    "tasks": 59
  },
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/lab/evaluation/",
    "json": "https://www.anchorterminal.com/lab/evaluation/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/lab/evaluation/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/lab/evaluation/index.min.md"
  },
  "markdown": "Each open-weight model the lab runs answers 59 questions about Anchor Terminal through our MCP server, pages and API, three times over, calling the tools itself or writing programs that call them, as it chooses. The right answers come from the site when each question is asked. The Lab score, 0 to 100, is the questions it got right with harder ones counting more.\n\nNo model has a Lab score yet.\n\n## Every evaluation\n\nA run in code mode, where the model may only write programs, is kept beside the Lab score's runs and scores nothing.\n\n| Date | Model | Mode | Score | Right |\n| --- | --- | --- | --- | --- |\n| 18 November 2026 | qwen3.8-27b | code only | 89 | 54 of 59 |\n\nThe questions: https://github.com/Anchor-Terminal/anchorlab/blob/main/eval/tasks.go\n\nAn answer is checked against the site before and after the model answered, a question whose subject left the site or that has two right answers is skipped, and an answer with no lookup behind it counts as wrong. A question counts as right when it's right in more than half of three passes; a level 2 question counts twice and a level 3 question three times.\n\nJSON: https://www.anchorterminal.com/api/v1/lab-evals.json\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-18",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "1.0",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "Benchmark",
        "url": "https://www.anchorterminal.com/benchmark/"
      },
      {
        "name": "Lab evaluation",
        "url": ""
      }
    ],
    "description": "Open-weight models answering 59 questions about Anchor Terminal through its MCP server, pages and API, scored 0 to 100, with every run dated.",
    "facts": [
      "59 questions",
      "three passes",
      "0 to 100"
    ],
    "h1": "Lab score: open-weight models on our own evaluation",
    "image": "https://www.anchorterminal.com/assets/og/benchmark-open-weights-runs.png",
    "path": "/lab/evaluation/",
    "published": "",
    "section": "benchmark",
    "title": "Lab score: open-weight models on our own evaluation | Anchor Terminal",
    "toc": null,
    "updated": "2026-11-18",
    "url": "https://www.anchorterminal.com/lab/evaluation/"
  },
  "tokens": {
    "markdown": 450,
    "slim": 330
  },
  "version": 1
}
