A machine that runs all night, and why the labs aren't at the forefront

The big labs, Anthropic, OpenAI and the others, might have the latest models that aren't available to us, and METR's report this week says Anthropic staff put their own speed-up from the most capable internal model at about 12 times in informal polls ^[1]. I don't think that puts them at the forefront of what can be done and is being done with agents. We, the people who work and build and test things out with agents, are at the forefront, because the labs might have better models but we're testing things that would take them a lot of time.

The harness moves the result more than the model

Zhang and colleagues tested this in May 2026 and found that on coding benchmarks "performance variance is governed more by harness configuration than by model choice" ^[2]. The same Claude model went from 45.9% to 55.4% on SWE-bench Pro when it moved from a standard scaffold to Claude Code, and in their own grid of three models and three harnesses, the harness moved scores 7.8 times as much as the model did ^[2]. Claude Code is Anthropic's own harness, but the model in that test was Claude Opus 4.5, which anyone could call through the API from November 2025 ^[3], so the jump came from something anyone with an API key could have built. You might not have the latest model, but agents, how you use them and what constraints you have can trigger much better results than the latest model.

The year of the leaderboard

Big companies, OpenAI among them, spent the spring trying the opposite, ranking engineers by tokens spent on formal or informal leaderboards ^[4]. Staff on Meta's internal dashboard used more than 60 trillion tokens in 30 days, the top user averaged 281 billion, which Fortune put at more than $1.4 million at the cheapest Claude Opus rate, and you could earn titles like "Token Legend" and "Cache Wizard" ^[5]. The dashboard came down two days after the news broke, and Meta told Fortune the employee who ran it took it down because its data had been shared outside the company ^[5].

By the end of May Fortune was calling tokenmaxxing over, and it lists Uber burning through its whole 2026 token budget in the first four months of the year, the Financial Times reporting Amazon employees running agents on pointless tasks to keep their numbers up for performance reviews, and Microsoft cancelling Claude Code for several product divisions, as The Verge reported ^[4]. Uber's COO said that if you can't draw a direct line to features shipped to users, the cost is "harder to justify" ^[4]. In August Microsoft's Jay Parikh told engineers that "Tokenmaxxing is not what we are optimizing for", as LeadDev reports from an email 404 Media saw, and Amazon retired its KiroRank dashboard after people started optimising for the usage itself ^[6].

In the survey for LeadDev's AI Impact Report, 58% of respondents said their organisation measures AI impact by token usage while 57% said token usage is a poor way to gauge real value, and only 19% rated tokenmaxxing as effective ^[6].

Loops are brute force

You can put a coding agent in a loop now, the same prompt fed back in every time it stops, and leave it running all night, and Anthropic turned that into an official Claude Code plugin in December 2025 ^[7]. You can have harnesses work all the time and get nothing useful at the end. Loops work when we have a thing to test against and we keep summarising and discovering and getting closer along the way, and DreamHost's write-up of the technique says that without clear acceptance tests "you'll just get a chaotic loop that invents things you didn't ask for" ^[7]. Loops aren't the end result, though. Loops are brute force, and there's something more elegant, I don't know what yet, but loops aren't clever.

Where the features come from

If you're building for people, the features they need are found by researching your market and talking to potential customers.

If you're building for agents, the features they need are found by testing the products and making requests cost efficient. On Friday night I asked for the token cost of our own MCP server to come down, and the structured answer to a search for "send email" went from 4,818 tokens to 3,481 once we dropped the links an agent can build from the slug and a note that repeated on every row ^[8]. Loading only the two tools Railway's MCP server needs for reading a service's logs costs about 1,900 tokens instead of about 52,000 for its whole list ^[9].

As far as I can tell, tokenmaxxing led to nothing useful. People call it slop because it's never been about quantity, it's always been about quality, and for quality products you still need a passionate human who knows the quirks to lead them. Tokenmaxxing was a trawler dragging a net across everything, and good products come from someone who knows the river and where the fish are.

Where ASI comes from

I think we're intelligent as a species because we started using tools, and our opposable thumb helps. Stout and Hecht, writing in PNAS in 2017, trace our culture back to "a uniquely human technological niche rooted in a shared primate heritage" of seeing and handling things ^[10]. The base model is the reasoning part, distilled from the internet, and tools for agents and agents are the answer to ASI, not the base model.

I think ASI won't be one of the big labs, it will be a group of people who figure out how to use the reasoning part and tool usage. Maybe agents will take a long time. Probably ASI is an agent framework, with LLMs as part of the solution and the rest made of the graph, the index, the embeddings and the other parts of the head, since that's the best we can imagine (one long-standing theory, from Teyler and DiScenna in 1986, has the hippocampus holding an index that gives "rapid and efficient access to memory content stored in neocortical areas" ^[11]). It will not look like this, but by building something similar to the brain we can slowly build towards it. Rich Sutton argued in 2019 that "general methods that leverage computation are ultimately the most effective, and by a large margin" ^[12], and if he's right the parts end up folded into a bigger model.

Tooling is the bottleneck, and why we exist

One of the most important bottlenecks is agent tooling, the APIs, MCP servers and services an agent has to find, sign up for, pay for and call, and that's why we exist. Of the 367 hosted MCP servers our graded listings name, 74 listed their tools when we asked on 6 November without credentials, 277 wanted a key or a sign-in first, and 16 didn't answer ^[13]. Resend's listing said its MCP server has 136 tools while seven of its panel reviews, written a month earlier, said 106 ^[14]. That's the kind of thing we're testing that would take the labs a lot of time.

We need more small, focused products, not big encompassing ones. So build cool things with agents and you'll be at the forefront.

References

Read on 7 November 2026. Quotes are as published.

1. [1]METR, 5 November 2026. Independent assessment of AI R&D acceleration at Anthropic. Source for the ≈12x self-reported speed-up from the most capable internal model, from Slack polls METR treats as loose evidence. metr.org
2. [2]Zhang, Y., Wang, J., Ge, Y., Xu, W., Hamm, J. and Reddy, C. K., 7 May 2026. Stop comparing LLM agents without disclosing the harness. arXiv 2605.23950. Source for the quote on harness configuration, the 45.9% and 55.4% on SWE-bench Pro, and harness-induced variance averaging 7.8 times model-induced variance. arxiv.org
3. [3]Anthropic, 24 November 2025. Introducing Claude Opus 4.5. Source for Opus 4.5 being available through the Claude API from that day. anthropic.com
4. [4]Fortune, Jeremy Kahn, 28 May 2026. Tokenmaxxing is over. Source for the leaderboards at Meta, Amazon and OpenAI, Uber spending its 2026 token budget in four months, the Financial Times report on Amazon employees, The Verge's report on Microsoft cancelling Claude Code subscriptions, and the Uber COO's quote, as Fortune reports them. fortune.com
5. [5]Fortune, 9 April 2026. Meta killed its employee AI token dashboard. Source for the 60 trillion tokens in 30 days, the top user's 281 billion and the $1.4 million estimate, the Token Legend and Cache Wizard titles, the dashboard coming down two days after the news, and Meta's statement. fortune.com
6. [6]LeadDev, Chantal Kapani, 17 August 2026. The tokenmaxxing hype didn't last long. Source for the AI Impact Report figures (58%, 57% and 19%), Jay Parikh's email as 404 Media reported it, and Amazon retiring KiroRank. leaddev.com
7. [7]DreamHost, 2026. The Ralph Wiggum loop, from first principles. Source for the loop that feeds the same prompt to a coding agent, Anthropic's official Claude Code plugin in December 2025, what it's good at and the quote on loops without acceptance tests. dreamhost.com
8. [8]Anchor Terminal measurements of its own MCP server against production data, 6 November 2026, not published. Source for the 4,818 and 3,481 tokens of the structured search answer.
9. [9]Anchor Terminal, 2026. The Railway MCP server listing, with the 71 tools its hosted endpoint lists. Source for the token counts for the whole list and for the two tools a log-reading job needs. anchorterminal.com
10. [10]Stout, D. and Hecht, E., 2017. Evolutionary neuroscience of cumulative culture. Proceedings of the National Academy of Sciences. Source for the quote on a uniquely human technological niche. evolutionaryneurosciencelab.hsites.harvard.edu
11. [11]Tanaka, K. Z. and McHugh, T. J., 2018. The hippocampal engram as a memory index. Journal of Experimental Neuroscience, volume 12. Source for Teyler and DiScenna's 1986 indexing theory and the quote on the hippocampus as an index to memory stored in the neocortex. pmc.ncbi.nlm.nih.gov
12. [12]Sutton, R., 13 March 2019. The bitter lesson. Source for the quote on general methods that leverage computation. incompleteideas.net
13. [13]Anchor Terminal research notes, 6 November 2026, not published. Source for the 367 hosted MCP servers named by graded listings and what each answered when asked without credentials.
14. [14]Anchor Terminal, 2026. Resend's listing, read on 6 November 2026. Source for the 136 tools its hosted MCP server lists and the 106 that seven of its panel reviews, written on 3 October 2026, report. anchorterminal.com