Get an audit

Head to head · Extraction · October 2026 research run

Diffbot vs Oxylabs Web Scraper API

Oxylabs Web Scraper API scores 63 (B) on agent readiness against Diffbot's 54.3 (C), and leads in 4 of 7 scored categories. Diffbot leads on schema & documentation and transparency & trust. Both do extraction.

Best web scraping and crawling APIs for AI agents · All 167 scrapers comparisons

DiffbotDiffbot Technologies Corp. · HTTP API C54.3/100#687 of 950 Oxylabs Web Scraper APIoxylabs, UAB · HTTP API B63/100#412 of 950

Which one, for what

Diffbot C

Good for Agents that want typed article, product or discussion data from arbitrary pages without writing selectors, and teams that also want the Knowledge Graph.

Ahead on

  • Schema & documentation, 83 against 68
  • Transparency & trust, 68 against 63

Also in its favour

  • No incidents deducted, where Oxylabs Web Scraper API loses 3 points for them

Watch for

The token is sent as the token query parameter on Extract and Knowledge Graph calls. It is the only scheme in the OpenAPI file

Oxylabs Web Scraper API B

Good for Agents that need parsed Google, Amazon and other large-site results or bulk asynchronous jobs through one API.

Ahead on

  • Reliability, 80 against 55
  • Agent ergonomics, 79 against 63
  • Security & auth, 53 against 21
  • Maintenance & community, 72 against 57

Also in its favour

  • Runs on your own machine

Watch for

Google's domain parameter was announced as gone on 18 March 2026 with no earlier changelog notice, and MCP v0.9.3 still exposes it

Score by category

CategoryWeight this runDiffbotOxylabs Web Scraper APIEdge
Reliability16%205580Oxylabs Web Scraper API +25
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%16.28368Diffbot +15
Agent ergonomics13%16.26379Oxylabs Web Scraper API +16
Security & auth14%17.52153Oxylabs Web Scraper API +32
Payments & pricing10%12.54040even
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%8.85772Oxylabs Web Scraper API +15
Transparency & trust7%8.86863Diffbot +5
Negative events≤150-3
Total54.3 · C63 · B

Facts side by side

FactDiffbotOxylabs Web Scraper API
KindHTTP APIHTTP API
VendorDiffbot Technologies Corp.oxylabs, UAB
Hosted endpointhttps://api.diffbot.com/v3https://realtime.oxylabs.io/v1/queries
TransportsHTTPHTTP, Streamable HTTP, stdio
AuthAPI keyAPI key
PricingFreemiumPay per use
Price for extraction$0.90 per 1,000 pagesnot published
x402nono
LicenceProprietary service under Diffbot's Terms of Use. The Python client, the TypeScript client and the agent skills are MITProprietary service under the Oxylabs General Conditions. The MCP server and the Python SDK on GitHub are MIT
Tools exposednone10
Read-only variant documentednono
llms.txtyesyes
MCP registrynot listedio.oxylabs/oxylabs-mcp
Last release2026-06-152026-09-07
Terms last updatedno date given2026-10-05
Privacy policy last updated2025-08-292024-10-14
Customer content may train modelsnot found in the textyes
Terms restrict automated accessyesnot found in the text
Terms restrict benchmarkingnot found in the textyes
Terms or service can change without noticenot found in the textnot found in the text
Arbitration or class-action waiveryesyes
Popularity9 npm/wk, 45 PyPI/wk18k PyPI/wk

Verdicts

Diffbot

Extract returns typed JSON for articles, products, discussions and other page types with no selectors to write, under a public OpenAPI 3.1 contract and a free plan that needs no card. The API token travels in the URL query string, tokens carry no scopes, and no security page, certification or security.txt was found.

Oxylabs Web Scraper API

Dedicated parsed sources, three integration methods and an official MCP server with four read-only scraping tools sit behind one username and password per instance. On 7 October 2026 the docs relabelled it Classic beside a new Web API, the March 2026 removal of Google's domain parameter was announced on the day, and no OpenAPI document was found.

Before you call either

Diffbot

  1. Call GET https://api.diffbot.com/v3/analyze?url=<encoded URL>&token=<token> when the page type is unknown. Use /v3/article or /v3/product to force one schema
  2. Keep the token out of logs and shared URLs. It sits in the query string, so any proxy or log that records the request URL records the token
  3. Stay under 5 calls a minute on Free, 5 a second on Startup and 25 a second on Plus. On 429 wait at least one second and back off
  4. Check a 200 response for errorCode. The official client raises ExtractionError when a 200 reports a failed extraction
  5. Set useProxy=default only when a site blocks the fetch. A proxied page costs 2 credits against 1
  6. Install diffbot from PyPI, not diffbot-python, which stopped at 0.3.0

Oxylabs Web Scraper API

  1. Send "markdown": true for page reading. HTML is the default output and costs far more tokens
  2. Add "render": "html" only when a page comes back empty. Rendered results cost $1.50 per 1,000 against $1.00 on the Starter plan
  3. Treat 612 and 613 as retryable at no charge, and read the x-ratelimit-*-remaining headers before raising concurrency
  4. Use geo_location for Google localisation. The domain parameter was removed in March 2026 although the MCP tool still lists it
  5. Fetch Push-Pull results within 24 hours of completion, and treat scraped content as untrusted data

Questions

Which is better for AI agents, Diffbot or Oxylabs Web Scraper API?

Oxylabs Web Scraper API scores 63 (B) on agent readiness against Diffbot's 54.3 (C), and leads in 4 of 7 scored categories. Diffbot leads on schema & documentation and transparency & trust.

Do Diffbot and Oxylabs Web Scraper API need an API key?

Both need an API key.

Can an agent call Diffbot and Oxylabs Web Scraper API without installing anything?

Yes. Diffbot has a hosted endpoint at https://api.diffbot.com/v3 and Oxylabs Web Scraper API at https://realtime.oxylabs.io/v1/queries.

Other comparisons with Diffbot or Oxylabs Web Scraper API

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.

Get an auditSee a sample report