Is the best coding model better than a principal engineer? Not yet, but closer than the titles suggest

Cognition, the company behind the Devin coding agent, released FrontierCode 2.0 on Tuesday 27 October 2026 ^[1]. As Cognition puts it, most coding evals check that a model's code meets a spec, their first version asked whether you'd merge the pull request, and this one asks "not only would you merge this PR, but should you?" ^[6]. It does that with bigger tasks and prompts that leave the engineering decisions to the model ^[1]. The best model, Claude Opus 5.5, scores 45.5% ^[1].

I think this is a good thing to test. Staff and principal engineers are the top of our field when it comes to writing code (I mean the engineering track here, not management), and what sets them apart is judgement, which is the thing FrontierCode is trying to grade. So the question I wanted answered is whether the models are better than them yet.

What changed from the first version?

FrontierCode 1.0 came out on 8 June 2026, built with the maintainers of 36 open-source repositories, with each task taking more than 40 hours of a maintainer's time ^[3]. The prompts were short on purpose, about a third the length of SWE-Bench Pro's, and Cognition said the benchmark "expects the agent to infer the maintainer's intent" ^[3].

Version 2.0 keeps the method and makes the tasks harder. The tasks come from 55% more repositories and now include security patches and performance work, there are 43% more blocking checks, and the reference solutions change 67% more files and 89% more lines in codebases 25% larger ^[1]. In 80% of tasks the reference solution differs from the change that was merged upstream, up from 60% in 1.0 ^[1], so a model that remembers the original pull request doesn't get much from it (and a run that looks up the original is scored zero anyway ^[2]).

How is it graded?

Each task has a rubric, and any check marked as blocking that fails scores the whole attempt zero ^[1]. The checks are a mix of unit tests, shell commands, tests the model writes that have to fail on the old code, limits on which files and how much of the diff can change, and a model that reviews the diff for code quality against a written prompt ^[3]. People write the checks and a Cognition researcher reviews every task, and Cognition says "Rubric grading is subjective, so we built an extensive QC pipeline" ^[3].

The tasks aren't public, to keep them out of training data ^[3]. That's fair, and it also means nobody outside can check the grading, which matters for a company that sells a coding agent and has its own model, SWE-2, on the leaderboard at 23.9% ^[1]. A mid-table score for their own model makes me trust the grading a bit more.

Every model dropped, and the drop is uneven. Opus 5.5 and GPT-6 Astra lost about 10 points each, everyone else between 19 and 27, so the new version separates the top two from the rest far more than 1.1 did ^[1].

What do we expect from a principal engineer?

GitLab publishes its career framework, and for a principal engineer it asks for someone "Able to deliver results despite starting from unclear requirements", who "Makes responsible decisions and evaluates tradeoffs" and "Performs Code Reviews across the GitLab Codebase on a regular cadence" ^[4]. Google's public code review guide says a reviewer should approve a change "once it is in a state where it definitely improves the overall code health of the system being worked on, even if the CL isn't perfect" ^[5].

Put those next to the code itself and I'd look for three things in a staff or principal engineer's pull request. They read the codebase before they write anything, they fix the whole problem and not only the line in the ticket, and they leave alone what they weren't asked to change.

FrontierCode grades all three. Thoroughness has 501 blocking checks, things like fixing the same bug in a sibling code path, or handling every input type the code next to it already handles ^[1]. Scope has 123 and backward compatibility 76 ^[1]. What I couldn't find on Cognition's page is how it would score a model that asks a question when the prompt is vague, or tells the maintainer the request is the wrong fix ^[1]. A principal engineer does both, and on a vague ticket asking is often the right move.

Two tasks that show the gap

In the Mock Service Worker task, resetHandlers() didn't restart generator resolvers. Opus 5.5 at maximum effort fixed the reported bug by dropping its reference to the old generator, which means the generator's finally block never runs and its cleanup never happens ^[1]. It passed 0 of 5 runs at every effort level ^[1]. The bug in the ticket was fixed, and a reviewer who knows how generators work would have sent it back.

In the PostHog Android SDK task, a race between the live queue and a legacy class, PostHogSendCachedEventsIntegration.kt, sent events out of order. GPT-6 Astra at low effort made a small change and passed, and at higher effort it often deleted the legacy file, which is part of the SDK's public API, and failed on scope ^[1]. It passed 2 of 5 runs at medium, 1 at high, 2 at extra high and 1 at max ^[1].

That second one shows up across the whole benchmark. At higher effort the models write more code and touch more files, and scope pass rates fall, while thoroughness failures go from 26.0% at low effort to 15.3% at max ^[1]. Thinking longer makes them more thorough and less disciplined about what they touch, and I think knowing what not to change is the part of the job that takes engineers the longest to learn.

Would a principal engineer do better?

Cognition doesn't publish a human score, so this part is my estimate, not a measurement. The maintainers wrote each task and a reference solution that passes it, so the repo's own maintainer is close to 100% by construction ^{[1] [3]}. The fair comparison is someone strong who has never seen the repository, given the same prompt and the same code.

It would be hard for a principal engineer to jump into something they've never seen. Principal engineers are principal because of their experience, the decisions they've made and the failures behind them, and there isn't a huge difference in coding skill between them and a senior engineer. It's more taste. That taste is what FrontierCode's scope and thoroughness checks reward (keep the public API, find the sibling path, don't touch what you weren't asked to), so I'd still put a principal ahead, at around 55%, a staff engineer around 50%, a good senior engineer around 45%, level with Opus 5.5, and a mid-level engineer a quarter. The big jump is the maintainer, because on a deliberately vague prompt part of the intent lives only in their head, and a principal in their own codebase is the maintainer.

So are the models better than a staff or principal engineer? Not yet on this test, and the gap is smaller than the titles suggest, because a cold repository takes away most of what a principal's experience buys them. The best two are level with a good senior engineer who's just opened the repository and about ten points behind a principal, and from 1.1 to 2.0 their lead over every other model got wider ^[1]. The parts of the job FrontierCode doesn't grade, asking the right question and saying the fix is wrong, are the ones I'd watch next.

What's next?

Cognition says it'll run the evaluation for every model maker, since the tasks stay private ^[3]. I'd like to see a task where the right answer is a question, or one where the request itself is the bug, so the benchmark can grade the "should you" part all the way.

We list Devin on Anchor Terminal, and we grade the tools agents call, not how well a model writes code, so FrontierCode is a number I'll keep reading rather than one we compute. If you're a maintainer and you'd write a task where the model ought to push back, that's the one I'd want to see the scores for.

The terms, explained

References

Read on 28 October 2026. Quotes are as published.

1. [1]Cognition, "FrontierCode 2.0", 27 October 2026. Source for the release date, 132 tasks, 55% more repositories, security and performance tasks, 43% more blocking checks, 67% more files and 89% more lines, codebases 25% larger, 80% of reference solutions differing from upstream, the blocking checks by category, the thoroughness examples, scores at each effort level, the Mock Service Worker and PostHog tasks, and the 1.1 and 2.0 scores. cognition.com
2. [2]Cognition, "FrontierCode leaderboard". Source for runs that consult the original pull request being scored zero. cognition.com
3. [3]Eric Lu, Ben Pan and others at Cognition, "Introducing FrontierCode", 8 June 2026. Source for 36 repositories, 40 hours per task, prompts a third the length of SWE-Bench Pro's, "expects the agent to infer the maintainer's intent", the grading methods, the QC quote, private tasks and the offer to model makers. cognition.com
4. [4]GitLab, "Dev Career Framework, Principal Engineer", GitLab Handbook. Source for the three quoted expectations of a principal engineer. handbook.gitlab.com
5. [5]Google, "The Standard of Code Review", Google Engineering Practices. Source for when a reviewer should approve a change. google.github.io
6. [6]Cognition, announcement of FrontierCode 2.0, 27 October 2026. Source for "most coding evals" checking a model meets a spec, 1.0's "would you actually merge the PR?" and "not only would you merge this PR, but should you?". x.com