{
  "data": {
    "author": "Vlad Cealicu",
    "faq": [
      {
        "a": "Real World Engineering Bench, Xplore's benchmark of 20 mechanical, electrical and aerospace engineering tasks, released on 27 October 2026. Each model works in a sandbox with CAD and simulation tools and hands in a design that a program checks against every requirement.",
        "q": "What is RWE-bench?"
      },
      {
        "a": "GPT-6 Astra, with a 37% average pass rate. Claude Opus 5.5 is 4 points above GPT-6.1 Sol and Astra is 12 points above it, and Sol is the cheapest of the top three at $10 a run.",
        "q": "Which model scored best?"
      },
      {
        "a": "By a deterministic program, with no model and no person judging. It checks the submitted design against every requirement in the brief and runs physics solvers where a requirement needs one, such as stress and fatigue on the crankshaft.",
        "q": "How is it graded?"
      },
      {
        "a": "Xplore doesn't publish a human score. My estimate is that someone with no engineering training passes none of the 20 tasks, a graduate a few in their own field, and a specialist roughly the third in their own domain, since every task needs CAD, meshing and solvers and has to meet every requirement.",
        "q": "How would a person do on RWE-bench?"
      },
      {
        "a": "A task counts as passed if any one of k runs at it passes. A higher k says how often a model gets there at all, a pass@1 score how often it gets there first time.",
        "q": "What does pass@k mean?"
      }
    ],
    "kicker": "Blog · Opinion · 27 October 2026",
    "lede": "Xplore released RWE-bench on 27 October 2026, 20 real engineering tasks where a crankshaft, a memory bus or a turbine vane gets checked by physics solvers and not by a model. The best of 11 models, GPT-6 Astra, passes 37%. Most people handed the same briefs would pass none, so I think that's a better result than the headline makes it sound.",
    "readingMinutes": 6,
    "words": 1271
  },
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/blog/rwe-bench-engineering",
    "json": "https://www.anchorterminal.com/blog/rwe-bench-engineering.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/blog/rwe-bench-engineering.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/blog/rwe-bench-engineering.min.md"
  },
  "markdown": "By Vlad Cealicu, 27 October 2026. 1,271 words, about 6 min to read.\n\nXplore, a startup that builds training environments for engineering work, released RWE-bench (Real World Engineering Bench) on Tuesday 27 October 2026 [[1]](#ref-1) [[2]](#ref-2). It's 20 tasks in mechanical, electrical and aerospace engineering, run on 11 frontier models, and the best of them, OpenAI's GPT-6 Astra, has an average pass rate of 37% [[1]](#ref-1) [[2]](#ref-2). Xplore's own summary is \"AI now beats almost every human at math and coding, but cannot reliably solve physical engineering problems\" [[1]](#ref-1).\n\n- 37%: GPT-6 Astra's average pass rate, the best of 11 models\n- 20: tasks across mechanical, electrical and aerospace engineering\n- $10: a run for GPT-6.1 Sol, about a sixth of what Astra and Claude Opus 5.5 cost\n\nRWE-bench as published on 27 October 2026. Sources [1] and [2].\n\n## What are the tasks?\n\nThey read like a week at an engineering job, not a quiz. One is a one-piece steel crankshaft for a three-cylinder diesel, designed in CAD with its oil drillings, that has to take full load at 2,400 rpm, stay under a mass limit and last millions of cycles [[1]](#ref-1). Another is routing a DDR3L memory interface on a production FPGA board, 71 nets out of a 484-ball BGA on an eight-layer board with nearly 300 parts already placed [[1]](#ref-1). A third is meshing the flow passage round a high-pressure turbine vane, running CFD until it converges and reporting the heat transfer along the surface [[1]](#ref-1).\n\nAn engineer on Xplore's team who works in that field wrote each task, from problems they've met in industry, along with a reference design built with the same tools the agents get [[1]](#ref-1). Each model gets a technical brief, a sandbox with CadQuery, Gmsh, CalculiX and Python, its own native harness at maximum reasoning effort and hours per attempt, and it can run its own checks before it hands anything in [[1]](#ref-1).\n\n## Why the grading is the interesting part\n\n\"Every run is graded automatically by a deterministic verifier, with no LLM judge and no human grading\" [[1]](#ref-1). A program checks the design against every requirement and runs physics solvers where it has to, so the crankshaft gets checked for being a valid solid, its clearances, mass, balance, proof stress, fatigue, stiffness and bearing loads [[1]](#ref-1).\n\nI think this is the part that matters. Code and maths got good fast because you can check the answer (the tests pass or they don't) and models learn best where the outcome can be checked, which is Xplore's argument too [[3]](#ref-3). Most engineering work has never had that kind of check outside a company's own tools and the heads of the people who do it, and Xplore puts it as \"What's missing is the data\" [[3]](#ref-3). It's also what they sell, environments built round those workflows, so the benchmark is a shop window as much as a test (fair enough, ours is our shop window too).\n\n## What do the scores say?\n\nXplore's page puts GPT-6.1 Sol at $10 a run and says only two models score higher, GPT-6 Astra by 12 points and Claude Opus 5.5 by 4, both at almost six times the cost [[1]](#ref-1). With Astra at 37% [[2]](#ref-2), that puts Opus 5.5 at 29%, Sol at 25% and the other eight below Sol. Xplore's heading for it is \"Spending more buys little\" [[1]](#ref-1).\n\nSix times the money buys 12 points, so spending more isn't where the next jump comes from. Twenty tasks is a small set though, and on an average over 20 tasks one task is worth 5 points, so I'd read the order as rough and the gap between Astra and everyone else as the real signal. The page also plots pass@k, where a task counts as passed if any of k runs passes, for k up to five [[1]](#ref-1).\n\n## How would a person off the street do?\n\nXplore doesn't give a human score, so this part is my estimate, not a measurement. Hand the same 20 briefs to someone with no engineering training, with the same sandbox and the same hours, and I think they pass none of them. Before you get to the engineering you have to write the part in CadQuery, mesh it in Gmsh and run a finite element or CFD solver, and a pass means every requirement holds, the proof stress, the fatigue life, the skew budget, not most of them [[1]](#ref-1). I'd fail every one of them myself, and I've spent my career writing software.\n\nA good engineering graduate who knows those tools might get a task or two in their own field. A working engineer gets the ones in their own domain, because the people who wrote the tasks built a reference design for each with the same tools the agents get [[1]](#ref-1). What none of them does is pass a crankshaft, a memory bus and a turbine vane in the same afternoon. My rough baseline is about 0% for a random person, a few percent for a graduate, and something like a third for a specialist who's strong in one of the three fields and lost in the other two.\n\nOne task, from the brief to a design that passes every check\n\n| Who | Time | Outcome |\n| --- | --- | --- |\n| GPT-6 Astra | a few hours a run | passes 37% of the 20 tasks |\n| A specialist, in their own field | about a week | passes the third in their field |\n| An engineering graduate | a month or more | passes a task or two |\n| Someone off the street | doesn't finish | passes none |\n\nTime on a log scale.\n\nPass rate on RWE-bench's 20 tasks, in per cent\n\n| Who | Pass rate |\n| --- | --- |\n| GPT-6 Astra | 37 |\n| A specialist, strong in one field | 33 |\n| An engineering graduate | 5 |\n| Someone off the street | 0 |\n\nEach series has a zero-based scale.\n\nAgainst that, a model that passes 37% across mechanical, electrical and aerospace in one run is a strong result, it's a generalist doing a bit of everyone's job. There's still a lot of work to do before it's the engineer you'd hand the turbine to, and that's the point of a benchmark that checks the physics.\n\n## What's next?\n\nXplore says harder tasks and more domains come next, biomechanical, civil and materials engineering [[1]](#ref-1), and their thesis ends with the bar going from a bearing to an engine to a spacecraft [[3]](#ref-3).\n\nCadQuery, Gmsh and CalculiX aren't on Anchor Terminal yet, because they're libraries you run yourself and not something an agent calls over the network. If engineering goes the way coding went, someone will put them behind an API or an MCP server, and then we'll list them and score them like everything else.\n\n## The terms, explained\n\n### What is RWE-bench?\n\nReal World Engineering Bench, Xplore's benchmark of 20 mechanical, electrical and aerospace engineering tasks, released on 27 October 2026. Each model works in a sandbox with CAD and simulation tools and hands in a design that a program checks against every requirement.\n\n### Which model scored best?\n\nGPT-6 Astra, with a 37% average pass rate. Claude Opus 5.5 is 4 points above GPT-6.1 Sol and Astra is 12 points above it, and Sol is the cheapest of the top three at $10 a run.\n\n### How is it graded?\n\nBy a deterministic program, with no model and no person judging. It checks the submitted design against every requirement in the brief and runs physics solvers where a requirement needs one, such as stress and fatigue on the crankshaft.\n\n### How would a person do on RWE-bench?\n\nXplore doesn't publish a human score. My estimate is that someone with no engineering training passes none of the 20 tasks, a graduate a few in their own field, and a specialist roughly the third in their own domain, since every task needs CAD, meshing and solvers and has to meet every requirement.\n\n### What does pass@k mean?\n\nA task counts as passed if any one of k runs at it passes. A higher k says how often a model gets there at all, a pass@1 score how often it gets there first time.\n\n---\n\nMore in [Opinion](https://www.anchorterminal.com/blog/opinion/index.md) · Older opinion [Slopcore will be a thing](https://www.anchorterminal.com/blog/slopcore.md) · Newer opinion [OpenAI's models say the Exponential Time Hypothesis is false](https://www.anchorterminal.com/blog/openai-three-results.md)\n\n- Older: [Slopcore will be a thing](https://www.anchorterminal.com/blog/slopcore.md)\n- Newer: [Can you run GLM-5.3-Flash locally? Yes, on one big machine](https://www.anchorterminal.com/blog/run-glm-5-3-flash-locally.md)\n- All posts: https://www.anchorterminal.com/blog/index.md\n\n## References\n\nRead on 27 October 2026. Quotes are as published.\n\n[1] Xplore, \"RWE-bench, Real World Engineering Bench\", 27 October 2026. Source for the 20 tasks and three domains, the crankshaft, DDR3L and turbine vane tasks, how tasks are written, the sandbox tools, the harness and time per attempt, the deterministic verifier and physics solvers, GPT-6.1 Sol's $10 a run, the 12 and 4 point gaps and six times the cost, \"Spending more buys little\", pass@k and the domains coming next. https://xplore.ai/rwe-bench\n\n[2] Ryan Merket, \"Xplore AI launches engineering benchmark, with GPT-6 Astra at 37%\", Runtime Wire, 27 October 2026. Source for the release date, 11 models and GPT-6 Astra's 37%. https://runtimewire.com/article/xplore-ai-rwe-bench-engineering-models\n\n[3] Xplore, \"Thesis\". Source for verifiable outcomes, \"What's missing is the data\", what Xplore builds and the bearing, engine and spacecraft line. https://xplore.ai/thesis\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-10-28",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.7",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "Blog",
        "url": "https://www.anchorterminal.com/blog/"
      },
      {
        "name": "Opinion",
        "url": "https://www.anchorterminal.com/blog/opinion/"
      },
      {
        "name": "RWE-bench",
        "url": ""
      }
    ],
    "description": "Xplore released RWE-bench on 27 October 2026, 20 mechanical, electrical and aerospace engineering tasks graded by a program that runs physics solvers, with no model judging. GPT-6 Astra leads at 37%. Vlad Cealicu on what the tasks are, how a person off the street would do on them, why the grading matters and what the cost per run says.",
    "facts": [
      "Vlad Cealicu",
      "27 October 2026"
    ],
    "h1": "AI passes 37% of real engineering tasks, most of us would pass none",
    "image": "https://www.anchorterminal.com/assets/og/blog-rwe-bench-engineering.png",
    "path": "/blog/rwe-bench-engineering",
    "published": "2026-10-27",
    "section": "blog",
    "title": "RWE-bench, 37% on real engineering tasks, and a human baseline | Anchor Terminal",
    "toc": null,
    "updated": "2026-10-27",
    "url": "https://www.anchorterminal.com/blog/rwe-bench-engineering"
  },
  "tokens": {
    "markdown": 2500,
    "slim": 1280
  },
  "version": 1
}