Blog · Opinion · 21 October 2026
The tools that still copy Oracle's benchmark ban
More than two decades after we got decent open-source databases, Oracle's licence still says you can't publish a benchmark of its software without asking. Some of the tools agents use today have the same idea in their terms. Here they are, by name, in their own words.
Preview This post is scheduled for 21 October 2026 and may change before then. It isn't on the blog yet.
By Vlad Cealicu1,707 words7 min read
It's a shock to me that Oracle's terms still survive today, more than two decades after we got decent open-source database software. Somehow, and I think mostly due to aggressive policies and contracts, Oracle is still alive, and I think time will deal with that. Some of its habits have outlived the reasons anyone had for them, though, and they turn up in newer products too, where I'd ask the same question I'd ask Oracle. Why would a good product need your permission before you say how good it is?
The practice I mean has a name. In 1982 David DeWitt, then an assistant professor, ran his Wisconsin Benchmark and showed Oracle's database performing poorly. Larry Ellison was displeased, and Oracle's licence grew a clause forbidding anyone to publish a benchmark of it without permission, the DeWitt clause [1]. Oracle's licence for its software downloads, the Oracle Technology Network License Agreement, still says you may not "disclose results of any Program benchmark tests without Oracle's prior consent" [2].
- 1982Closed
David DeWitt's Wisconsin Benchmark shows Oracle's database performing poorly [1]
- Oracle's licenceClosed
The DeWitt clause forbids publishing benchmarks Oracle hasn't sanctioned [1]
- 16 October 2026Closed
Oracle's download licence still forbids disclosing benchmark results without its consent [2]
- Open to test
- Closed
In the meantime PostgreSQL has been PostgreSQL since 1996 and MySQL's first version appeared on 23 May 1995 [3] [4]. Anyone can benchmark either of them and publish what they find, which is how good software should want it. MySQL is developed by Oracle today [4].
How many of the tools agents use do this?
We read the terms of every vendor we list, and one of the things we record is whether they restrict benchmarking [5]. Of the 674 vendors whose terms we've read, 315 restrict benchmarking or using the service to build a competitor. That's almost half, the same share Quynh found when she counted terms documents rather than vendors. 107 of them restrict benchmarking or publishing performance results by name, and 28 forbid you to publish the results, which is Oracle's clause in substance [5].
Vendors
- Other clauses, mostly about building a competitor208
- Ban benchmarking the service, in one form or another69
- Forbid publishing the results28
- Only restrict benchmarking for competitive purposes9
- Allow publishing if the test can be repeated1
Bars start at zero. Values are labelled in each panel.
The clause isn't spread evenly. By the area a vendor's tools sit in, content generation and work apps restrict benchmarking or competing most often, more than half of their vendors, and communication and design tools least [5].
Per cent
- Content generation68
- Work & business apps59
- Voice & speech54
- Models & inference52
- Agent runtime51
- Domain data49
- Agent frameworks46
- Developer & infrastructure43
- Web, data & documents42
- Everyday APIs41
- Payments & protocols39
- Design & diagrams35
- Communication25
Bars start at zero. Values are labelled in each panel.
Who forbids you to publish the results?
These are the closest to Oracle. Their terms stop you telling anyone what you found without asking first.
| Vendor | What their terms say |
|---|---|
| 1upHealth | You may not "Perform security, benchmark, or penetration testing of the Services … or disclose the results of such testing" without its Chief Security Officer's consent |
| Airwallex | You may not "publicly disseminate performance information or analysis relating to the other Party's APIs" (the clause binds both sides) |
| Alchemer | You agree not to "disclose to third parties (whether in writing or orally) any benchmark test data related to the SaaS" (as we read it on 8 October) |
| Apollo | You may not "Disclose the results of any Platform or program benchmark tests to any third parties without Apollo's prior written consent." |
| Braintrust | You may not "perform or disclose … any benchmark, comparative test, performance or availability test, or evaluation of the Services or any results therefrom" |
| Buffer | You may not "conduct any benchmark or stress tests … or publish any performance data relating to the API or the Buffer Service" |
| CleverTap | You may not "publicly disseminate information regarding the performance of the Services" |
| Factorial | "publication of any unauthorised benchmark test results" is prohibited |
| Formbricks | You may not "publicly disseminate any performance data or analysis (including, but not limited to, benchmarks) related to the Software or Documentation, regardless of its origin" |
| GrowthBook | You may not "publicly disseminate benchmarks or performance information about the Services" |
| Intercom | You may not "publicly disseminate information regarding the performance of the Services" |
| LangSmith | You may not "publish or disclose benchmarks, performance results, or comparative information regarding the LangSmith Platform" |
| Luma AI | You may not "publish benchmarks or performance information about the Services" |
| Miro | You may not "publish benchmarks or performance information about the Service" |
| Omnisend | You may not "perform or publish any performance or benchmark tests or analyses relating to the Services" |
| Plaid | You may not "publish any benchmark or performance information about the Platform" |
| PlayHT | You may not "publish benchmarks or performance information about the Service" (from an archived copy of its terms) |
| PostHog | You may not "Publish benchmarking results about PostHog without our permission." |
| PubNub | You may not "disclose or publish, without PubNub's express prior written consent, performance or capacity statistics or the results of any benchmark test" |
| SambaNova | Not "without our prior written consent, publish or disclose information or results relating to capabilities, performance, performance comparisons or other “benchmarking” activities" |
| Secoda | You may not "disclose to any third party the results of any benchmark tests or other evaluation of the Secoda Services" |
| Sentry | You may not "publish benchmarks or performance information about the Service" |
| Stadia Maps | You may not "publish or communicate in any way a competitive analysis (such as benchmarking) of imagery" (its satellite imagery terms) |
| Terra | You may not "publicly disseminate information from any source regarding the performance of the Services or Output" |
| Xero | You may not use its developer platform "in order to share performance information about the developer platform (including uptime, response time, and benchmarks)" |
| Zep | You may not "engage in competitive analysis or benchmarking for public or third-party disclosure" |
| Zeplin | You may not "perform, publish, or disclose to third parties any evaluation or benchmark tests or analyses relating to the Services or use thereof without Zeplin's prior written consent" |
| ZoomInfo | You may not "disclose the results of any ZoomInfo Technology or program benchmark tests to any third parties without ZoomInfo's prior written consent" |
Apollo's and ZoomInfo's sentences are Oracle's almost word for word, with their own names in place of Oracle's. LangSmith is LangChain's platform for testing and evaluating agents, and its terms stop you publishing how it performs. Every quote is from the vendor's own terms, linked from its listing, as we read them in October 2026 [5].
Who forbids you to benchmark at all?
These go further than Oracle. The terms don't just stop you publishing, they stop you running the test.
Attio, Coinbase Developer Platform, Epsilla, Prismic, Salt Edge, Ship24, TableCheck, Turnkey and Voximplant ban benchmarking the service. Aembit, Amplitude, AssemblyAI, BigCommerce, Canva, Cline, Cohere, Commerce Layer, Copper, Crowdin, Deepgram, Descope, Documenso, Eat App, Everee, Explorium, Factory, FreeAgent, Freshworks, Front, Galileo, HeyGen, Instinct, Kustomer, Lambda, Lever, LiteAPI, LlamaParse, Lokalise, Manatal, Melius, Mem0, Microsoft, Nebius, OpenDocRouter, PagerDuty, Paid, Phrase, Pingram, Poke, Resemble AI, Retell AI, Sanity, Sprout Social, Statsig, TaxCloud, Together AI, Typeform, Validic, Weaviate, Workato, Zoho and Zoom ban it for "benchmarking or competitive purposes", for "competitive analysis or benchmarking", or in another wording that puts the two together. Moz, Sendcloud, Fabric and Brex ban benchmarking them against other products, which is the comparison an agent needs most. DeepL and SYSTRAN forbid "benchmark" or other "capacity" tests of their infrastructure, which is more about load than about results [5].
Many of these share one template sentence, against accessing the service to monitor its "availability, performance or functionality", often followed by some version of "or for any other benchmarking or competitive purposes". 24 vendors' terms carry it [5]. Cohere's is in capitals. Coinbase is one of the companies behind x402, the protocol for agents paying for tools without a human [6], and its developer terms, as we read them on 8 October, forbid using its tools "for purposes of monitoring the availability, performance, or functionality" of its products "or for any other benchmarking purposes" [5].
And nine, HelpDesk, Kintsugi, LandingAI, LeadMagic, LiveKit, Merge, Nutshell, Paragon and Wrike, only restrict benchmarking by competitors or for competitive purposes, which is the one version of this I understand. Cursor lets you publish your results as long as you include what others need to repeat the test, and that's the rule I'd want everyone to have [5].
We met the same thing this week. Photon, which we listed on 16 October, forbids publishing benchmarks of its service in its terms [7].
Why does it matter for agents?
An agent picks a tool on evidence, and the evidence is benchmarks, uptime and other people's experience. A clause that makes publishing that evidence a breach of contract is a clause against the agent's ability to choose. I think a vendor that's confident in its product wants the test run and the result published. I'll let you draw your own conclusions about the rest.
Some have ordinary reasons. DeepL and SYSTRAN are protecting shared infrastructure from load tests, 1upHealth runs a health data platform and doesn't want security testing it hasn't agreed to, and Stadia's clause sits in the terms for its satellite imagery. None of those needs a ban on publishing an honest result.
The kindest reading of the rest is that a lot of these were pasted from a template and never revisited, and 24 copies of the same sentence suggest that's true for many. If you're one of them, deleting the clause costs you nothing. If you meant it, I'd like to hear why.
We grade from public evidence and haven't accepted any of these terms, so none of them binds our grades, and the same goes for the uptime checks we plan to run. If we ever sign up to one of these tools to run an audit, we'll ask for written permission first, and say on the listing if we were refused. Our own terms don't restrict benchmarking Anchor Terminal, and you're welcome to.
References
Read on 16 and 17 October 2026.
- [1]Wikipedia, David DeWitt. Source for the 1982 Wisconsin Benchmark study, Larry Ellison's displeasure and Oracle establishing the original DeWitt clause. en.wikipedia.org
- [2]Oracle, Oracle Technology Network License Agreement, last updated 9 September 2014, the licence for Oracle's software downloads. Source for the clause forbidding disclosure of Program benchmark test results without Oracle's prior consent. oracle.com
- [3]Wikipedia, PostgreSQL. Source for the name PostgreSQL in use since 1996. en.wikipedia.org
- [4]Wikipedia, MySQL. Source for the first internal release on 23 May 1995 and Oracle as its developer today. en.wikipedia.org
- [5]Anchor Terminal, vendors' terms as read by our policy reader, 8 to 17 October 2026. A clause counts whether our rules found it or a reader added it after reading the document, as on each listing. Every quote was rechecked against the live page on 16 or 17 October except Coinbase's and Alchemer's, which refused automated readers that day, and PlayHT's, which we could only read from an archived copy. Source for the counts, the shares by area (a vendor counts in every area its listings sit in, and every area has at least 15 vendors) and every quoted clause. Atlassian's two products share one agreement and count once. Vast.ai, Soniox, HERE and Apiroc use the word benchmark in another sense, so they aren't counted as restricting benchmarking by name, though other clauses of theirs put them among the 315. Each listing links the terms page it was read from. anchorterminal.com
- [6]Anchor Terminal, x402. Source for Coinbase among the companies on or supporting the protocol. anchorterminal.com
- [7]Anchor Terminal, the Photon listing, 16 October 2026. Source for Photon's terms forbidding publication of benchmarks. anchorterminal.com