Blog · Opinion · 27 October 2026
AI passes 37% of real engineering tasks, most of us would pass none
Xplore released RWE-bench on 27 October 2026, 20 real engineering tasks where a crankshaft, a memory bus or a turbine vane gets checked by physics solvers and not by a model. The best of 11 models, GPT-6 Astra, passes 37%. Most people handed the same briefs would pass none, so I think that's a better result than the headline makes it sound.
By Vlad Cealicu1,271 words6 min read
Xplore, a startup that builds training environments for engineering work, released RWE-bench (Real World Engineering Bench) on Tuesday 27 October 2026 [1] [2]. It's 20 tasks in mechanical, electrical and aerospace engineering, run on 11 frontier models, and the best of them, OpenAI's GPT-6 Astra, has an average pass rate of 37% [1] [2]. Xplore's own summary is "AI now beats almost every human at math and coding, but cannot reliably solve physical engineering problems" [1].
What are the tasks?
They read like a week at an engineering job, not a quiz. One is a one-piece steel crankshaft for a three-cylinder diesel, designed in CAD with its oil drillings, that has to take full load at 2,400 rpm, stay under a mass limit and last millions of cycles [1]. Another is routing a DDR3L memory interface on a production FPGA board, 71 nets out of a 484-ball BGA on an eight-layer board with nearly 300 parts already placed [1]. A third is meshing the flow passage round a high-pressure turbine vane, running CFD until it converges and reporting the heat transfer along the surface [1].
An engineer on Xplore's team who works in that field wrote each task, from problems they've met in industry, along with a reference design built with the same tools the agents get [1]. Each model gets a technical brief, a sandbox with CadQuery, Gmsh, CalculiX and Python, its own native harness at maximum reasoning effort and hours per attempt, and it can run its own checks before it hands anything in [1].
Why the grading is the interesting part
"Every run is graded automatically by a deterministic verifier, with no LLM judge and no human grading" [1]. A program checks the design against every requirement and runs physics solvers where it has to, so the crankshaft gets checked for being a valid solid, its clearances, mass, balance, proof stress, fatigue, stiffness and bearing loads [1].
I think this is the part that matters. Code and maths got good fast because you can check the answer (the tests pass or they don't) and models learn best where the outcome can be checked, which is Xplore's argument too [3]. Most engineering work has never had that kind of check outside a company's own tools and the heads of the people who do it, and Xplore puts it as "What's missing is the data" [3]. It's also what they sell, environments built round those workflows, so the benchmark is a shop window as much as a test (fair enough, ours is our shop window too).
What do the scores say?
Xplore's page puts GPT-6.1 Sol at $10 a run and says only two models score higher, GPT-6 Astra by 12 points and Claude Opus 5.5 by 4, both at almost six times the cost [1]. With Astra at 37% [2], that puts Opus 5.5 at 29%, Sol at 25% and the other eight below Sol. Xplore's heading for it is "Spending more buys little" [1].
Six times the money buys 12 points, so spending more isn't where the next jump comes from. Twenty tasks is a small set though, and on an average over 20 tasks one task is worth 5 points, so I'd read the order as rough and the gap between Astra and everyone else as the real signal. The page also plots pass@k, where a task counts as passed if any of k runs passes, for k up to five [1].
How would a person off the street do?
Xplore doesn't give a human score, so this part is my estimate, not a measurement. Hand the same 20 briefs to someone with no engineering training, with the same sandbox and the same hours, and I think they pass none of them. Before you get to the engineering you have to write the part in CadQuery, mesh it in Gmsh and run a finite element or CFD solver, and a pass means every requirement holds, the proof stress, the fatigue life, the skew budget, not most of them [1]. I'd fail every one of them myself, and I've spent my career writing software.
A good engineering graduate who knows those tools might get a task or two in their own field. A working engineer gets the ones in their own domain, because the people who wrote the tasks built a reference design for each with the same tools the agents get [1]. What none of them does is pass a crankshaft, a memory bus and a turbine vane in the same afternoon. My rough baseline is about 0% for a random person, a few percent for a graduate, and something like a third for a specialist who's strong in one of the three fields and lost in the other two.
- GPT-6 Astraa few hours a run
passes 37% of the 20 tasks
- A specialist, in their own fieldabout a week
passes the third in their field
- An engineering graduatea month or more
passes a task or two
- Someone off the streetdoesn't finish
passes none
Bar length is time on a log scale, and the bars grow at the same speed. A hatched bar doesn't finish.
Pass rate
- GPT-6 Astra37
- A specialist, strong in one field33
- An engineering graduate5
- Someone off the street0
Bars start at zero. Values are labelled in each panel.
Against that, a model that passes 37% across mechanical, electrical and aerospace in one run is a strong result, it's a generalist doing a bit of everyone's job. There's still a lot of work to do before it's the engineer you'd hand the turbine to, and that's the point of a benchmark that checks the physics.
What's next?
Xplore says harder tasks and more domains come next, biomechanical, civil and materials engineering [1], and their thesis ends with the bar going from a bearing to an engine to a spacecraft [3].
CadQuery, Gmsh and CalculiX aren't on Anchor Terminal yet, because they're libraries you run yourself and not something an agent calls over the network. If engineering goes the way coding went, someone will put them behind an API or an MCP server, and then we'll list them and score them like everything else.
The terms, explained
What is RWE-bench?
Real World Engineering Bench, Xplore's benchmark of 20 mechanical, electrical and aerospace engineering tasks, released on 27 October 2026. Each model works in a sandbox with CAD and simulation tools and hands in a design that a program checks against every requirement.
Which model scored best?
GPT-6 Astra, with a 37% average pass rate. Claude Opus 5.5 is 4 points above GPT-6.1 Sol and Astra is 12 points above it, and Sol is the cheapest of the top three at $10 a run.
How is it graded?
By a deterministic program, with no model and no person judging. It checks the submitted design against every requirement in the brief and runs physics solvers where a requirement needs one, such as stress and fatigue on the crankshaft.
How would a person do on RWE-bench?
Xplore doesn't publish a human score. My estimate is that someone with no engineering training passes none of the 20 tasks, a graduate a few in their own field, and a specialist roughly the third in their own domain, since every task needs CAD, meshing and solvers and has to meet every requirement.
What does pass@k mean?
A task counts as passed if any one of k runs at it passes. A higher k says how often a model gets there at all, a pass@1 score how often it gets there first time.
References
Read on 27 October 2026. Quotes are as published.
- [1]Xplore, "RWE-bench, Real World Engineering Bench", 27 October 2026. Source for the 20 tasks and three domains, the crankshaft, DDR3L and turbine vane tasks, how tasks are written, the sandbox tools, the harness and time per attempt, the deterministic verifier and physics solvers, GPT-6.1 Sol's $10 a run, the 12 and 4 point gaps and six times the cost, "Spending more buys little", pass@k and the domains coming next. xplore.ai
- [2]Ryan Merket, "Xplore AI launches engineering benchmark, with GPT-6 Astra at 37%", Runtime Wire, 27 October 2026. Source for the release date, 11 models and GPT-6 Astra's 37%. runtimewire.com
- [3]Xplore, "Thesis". Source for verifiable outcomes, "What's missing is the data", what Xplore builds and the bearing, engine and spacecraft line. xplore.ai