# Crawl4AI vs Diffbot (slim)

> Diffbot scores 56.1 (C) to Crawl4AI's 43.9 (E) for scraping. Prices, MCP, x402, uptime and agent notes side by side.

- Full: https://www.anchorterminal.com/compare/crawl4ai-vs-diffbot.md (~5,500 tokens) · this version ~2,830 tokens · JSON https://www.anchorterminal.com/compare/crawl4ai-vs-diffbot.json · canonical https://www.anchorterminal.com/compare/crawl4ai-vs-diffbot
- Index: https://www.anchorterminal.com/llms.txt · API: https://www.anchorterminal.com/api/v1/index.json · MCP: https://www.anchorterminal.com/mcp · Updated: 2026-11-07
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)

Diffbot scores 56.1 (C) on agent readiness against Crawl4AI's 43.9 (E), and leads in 2 of 7 scored categories. Crawl4AI leads on security & auth, payments & pricing and maintenance & community. Both do scraping.

- Crawl4AI: E 43.9, rank #1224 of 1486 · https://www.anchorterminal.com/tools/crawl4ai.min.md
- Diffbot: C 56.1, rank #840 of 1486 · https://www.anchorterminal.com/tools/diffbot.min.md
- Crawl4AI, good for Agents that need cheap page-to-Markdown at volume, Google results and search in one key, or a free self-hosted Python crawler. Ahead on Security & auth, 46 against 21; Payments & pricing, 45 against 40; Maintenance & community, 83 against 77. Also Open source.
- Diffbot, good for Agents that want typed article, product or discussion data from arbitrary pages without writing selectors, and teams that also want the Knowledge Graph. Ahead on Reliability, 55 against 35; Schema & documentation, 83 against 55. Also No incidents deducted, where Crawl4AI loses 10 points for them.

| Category | Crawl4AI | Diffbot |
| --- | --- | --- |
| Reliability (16%) | 35 | 55 |
| Performance (10%) | pending | pending |
| Schema & documentation (13%) | 55 | 83 |
| Agent ergonomics (13%) | 67 | 63 |
| Security & auth (14%) | 46 | 21 |
| Payments & pricing (10%) | 45 | 40 |
| Task success (10%) | pending | pending |
| Maintenance & community (7%) | 83 | 77 |
| Transparency & trust (7%) | 70 | 68 |

| Fact (where they differ) | Crawl4AI | Diffbot |
| --- | --- | --- |
| Vendor | CONTEXT4AI PTE LTD | Diffbot Technologies Corp. |
| Hosted endpoint | `https://api.crawl4ai.com/mcp` | `https://api.diffbot.com/v3` |
| Transports | Streamable HTTP | HTTP |
| Price for scraping | $0.50 per 1,000 pages | not published |
| Licence | Apache-2.0 | Proprietary service under Diffbot's Terms of Use. The Python client, the TypeScript client and the agent skills are MIT |
| Tools exposed | 16 | none |
| Last release | 2026-10-14 | 2026-10-29 |
| Terms last updated | 2026-10-15 | no date given |
| Privacy policy last updated | 2026-10-15 | 2025-08-29 |
| Terms restrict automated access | not found in the text | yes |
| Arbitration or class-action waiver | not found in the text | yes |
| Popularity | 87k stars, 368k PyPI/wk | 9 npm/wk |

Why the edge:
- Schema & documentation, Diffbot +28: Crawl4AI 55 (No OpenAPI or similar contract for the cloud API, and the MCP tool schemas are only in the live tools/list, which needs a key. /v1/prices is machine-readable but covers prices, not calls (0). llms.txt of 304 lines written for agents (10). Each endpoint says what it is for and when to choose another, such as trying without a proxy first, using `browser` only when `auto` misses content and /scrape/jobs for slow lists (16 of 20). Parameters are listed with types, defaults, allowed values (`format` both, md or html; `proxy` none, isp or residential) and limits, in prose (10 of 15). Curl examples per endpoint and documented error strings per status with whether each is charged (14 of 15). No changelog for the cloud API, only dated notes such as /serp 'added 8 Oct 2026'. The library has a dated CHANGELOG (5 of 15). 0 + 10 + 16 + 10 + 14 + 5 = 55.); Diffbot 83 (The Extract APIs have a public OpenAPI 3.1.0 file, version 1.1.0, with 13 operations, and the reference pages are generated from it. We read that file and one rendered page. The descriptions for Crawl, Knowledge Graph and Web Search were not read (25). llms.txt exists but is a short index with no Markdown twins of the docs pages found (5 of 10). The Analyze API page has sections on when to use it and when not to (17 of 20). Parameters are typed, `url` is the one required field and the file has 15 enums, but `fields` is a comma-separated string and the file defines no shared response schemas (10 of 15). The file carries 24 examples. It documents only 200 and 500, with separate docs pages for 401, 404, 429 and 457 (11 of 15). The path is versioned `/v3` and a dated changelog with RSS runs from 2015 to 15 June 2026 (15). All per the 9 October check, not re-read on 4 November (terms bar automated access to the site). 25 + 5 + 17 + 10 + 11 + 15 = 83.)
- Security & auth, Crawl4AI +25: Crawl4AI 46 (Bearer keys, shown once and regenerable, with 3 to 30 keys per plan. claude.ai and ChatGPT sign in through a flow that advertises one scope, `mcp`, and each app shows as a revocable key. No per-key scopes, and the docs don't offer the key in a URL (22 of 30). A key can call billing routes, and the MCP `topup` and `recharge` tools charge the saved card, though llms.txt tells agents to ask first and accounts can set a spend cap. No read-only key (6 of 20). Scraped pages are untrusted text and no prompt-injection guidance was found in llms.txt or the API docs (0). Per-call cost, balance and fetch path in response headers and fields, no access log documented (6 of 15). The security page (4 October 2026) gives a disclosure process with fix targets and says SOC 2 is in progress, and the library publishes GitHub advisories with reporter credits. security.txt redirects to a 404, and no bounty found (12 of 20). 22 + 6 + 0 + 6 + 12 = 46.); Diffbot 21 (One token per account, sent in the URL query string as the documented and only scheme in the OpenAPI file. Tokens are revocable in the dashboard per a June 2025 changelog title, and child tokens exist on Plus and above (20 of 30, less 10 for the secret in the URL, so 10). Child tokens have the same rights as the primary one and there is no read-only mode. The skills limit the agent to the `db` binary through `allowed-tools` (5 of 20). The API returns untrusted web content and no guidance on prompt injection was found (0 of 15). The token details page shows usage activity and an Account API reports usage. No per-call log was found (6 of 15). No security.txt, security page, certification or bug bounty was found on the pages read, and the privacy policy says some API calls rely on HTTP without encryption (0 of 20). Site pages per the 9 October check, not re-read on 4 November. OSV lists no advisory for `diffbot` on PyPI or `@diffbot/typescript` on npm on 4 November. 10 + 5 + 0 + 6 + 0 = 21.)
- Reliability, Diffbot +20: Crawl4AI 35 (Graded on the hosted lines, for Crawl4AI Cloud, which the README and llms.txt now lead with and which an agent calls through its MCP server; the library would read on the local lines. No status page is linked, and status.crawl4ai.com fails TLS (0). No incident history readable (5). Per-plan limits are published with numbers in /v1/prices, from 10 requests a minute and 2 at a time on Free to 1,500 and 50 on Scale (15). 429 carries X-RateLimit headers and a message, 503 busy says to resend after a few seconds, and MCP job submissions take an idempotency key, but no Retry-After or backoff guidance (10 of 15). No SLA found in the terms or pricing (0). Prices and the free credit are a soft launch, /serp is beta and /answer experimental, while scrape, extract and search carry no beta label (5 of 10). 0 + 5 + 15 + 10 + 0 + 5 = 35.); Diffbot 55 (Graded as a hosted API. status.diffbot.com is a Pingdom public report with 13 checks, among them Analyze API, Article API, Crawlbot and Websearch, with daily uptime for seven days and no incident notes (15 of 20). No 90-day record can be read. In the week to 9 October 2026 Analyze API shows 99.69 per cent, with 2, 11 and 17 minutes down on 4, 5 and 6 October, and Article API 5, 9 and 2 minutes on 5 to 7 October (5 of 30, the line for no readable history). Rate limits are published per plan, 5 calls a minute on Free, 5 a second on Startup and 25 a second on Plus (15). The 429 page says to pause one second and back off. No `Retry-After` header is documented, though the official client reads one (10 of 15). No SLA was found (0). The Extract APIs are at `/v3` with no beta label (10). All per the 9 October check, not re-read on 4 November: Diffbot's terms bar software that scrapes or crawls the Site or Service, so its site and status report weren't fetched. 15 + 5 + 15 + 10 + 0 + 10 = 55.)
- Maintenance & community, Crawl4AI +6: Crawl4AI 83 (Library v0.9.5 on 14 October 2026, 23 days before 6 November, and the cloud added /serp on 8 October (30). v0.9.3 (31 August), v0.9.4 (23 September) and v0.9.5 (14 October) in the 90 days from 8 August (20). 56 open issues on 6 November, the newest triaged with Bug, Needs Triage, In-progress, Root caused or Done labels within days, plus a Discord. Reply authors weren't read (20 of 25). Not in the official MCP registry, and the cloud has no official SDK; the Python library is current on PyPI (8 of 15). The security workflow runs a test suite and a default-posture gate, and releases are built and checked with twine. No general test workflow on push found (5 of 10). 30 + 20 + 20 + 8 + 5 = 83.); Diffbot 77 (Every dated source of the vendor's own was looked for in its GitHub repositories, since its site wasn't re-read. The official Python client, `diffbot` in the listing's packages, was tagged v3.1.0 on 29 October 2026, adding a Web Search `snippet_length` option, and the agent skills were tagged v1.2.0 the same day with rewritten skill content, 6 days before this check (30; was 10 on the API changelog's 15 June 2026 entry, which is per the 9 October check, not re-read on 4 November). In the 90 days from 6 August, the skills were tagged v1.1.0 (12 August, five new skills), v1.1.1 (13 August, a plugin marketplace manifest, counted half) and v1.2.0, and the Python client v3.0.0 (28 September) and v3.1.0, counted half as SDK tags, 3.5 in all (20). A public changelog with RSS and a support address exist. GitHub issues were not read (8 of 15 for a closed service). Python and TypeScript clients exist, but the docs named the packages `diffbot-python` and `diffbot-typescript` on 9 October while the repositories publish `diffbot` and `@diffbot/typescript`, and the TypeScript repository has no tag and no commit since 17 August 2026 (12 of 15). The Python repository tests on Python 3.9 to 3.13. We did not see CI results (7 of 10). 30 + 20 + 8 + 12 + 7 = 77.)
- Payments & pricing, Crawl4AI +5: Crawl4AI 45 (No x402, MPP or L402 (0). Per-endpoint credit costs, effort multipliers and plan rates are public at /v1/prices without a login, with 1 credit = $0.001 (20). 1,000 free credits a month, no card (20). A 7-day pass key comes from a click on api.crawl4ai.com without an account, which is still a person in a browser, and a permanent key needs email verification (5 of 20). 0 + 20 + 20 + 5 = 45.); Diffbot 40 (No x402, MPP or L402 was found (0 of 40). Plan prices, the credit cost of each action and overage per credit are public without a login (20). The Free plan has 10,000 credits a month and needs no card (20). A person signs up in a browser at app.diffbot.com to get a token (0). Pricing per the 9 October check, not re-read on 4 November. 0 + 20 + 20 + 0 = 40.)