{ "data": { "a": { "slug": "crw", "name": "fastCRW", "vendor": "Nanofy OÜ", "vendorUrl": "https://fastcrw.com", "kind": "http-api", "category": "web-scrapers", "summary": "fastCRW is a Firecrawl-compatible scraping engine in Rust from Nanofy OÜ in Estonia. It turns pages into Markdown or JSON and searches, maps and crawls sites, as an AGPL-3.0 self-hosted binary or a hosted API with an MCP server.", "url": "https://www.anchorterminal.com/tools/crw", "markdownUrl": "https://www.anchorterminal.com/tools/crw.md", "slimMarkdownUrl": "https://www.anchorterminal.com/tools/crw.min.md", "jsonUrl": "https://www.anchorterminal.com/api/v1/tools/crw.json", "dns": "crw.grade.anchorterminal.com", "repo": "https://github.com/fastcrw/crw", "license": "AGPL-3.0", "transports": [ "stdio", "streamable-http" ], "remoteUrl": "https://fastcrw.com/mcp", "packages": [ { "registry": "npm", "name": "crw-mcp" }, { "registry": "npm", "name": "crw-sdk" }, { "registry": "pypi", "name": "crw" }, { "registry": "cargo", "name": "crw-server" }, { "registry": "oci", "name": "ghcr.io/fastcrw/crw" } ], "auth": "mixed", "authNotes": "Hosted API at api.fastcrw.com takes `Authorization: Bearer crw_live_...` keys from the fastcrw.com dashboard after a browser signup (GitHub or email). The hosted MCP server at fastcrw.com/mcp takes OAuth 2.1 (PKCE S256, dynamic client registration, one scope `mcp:tools`) or the key in the URL path as fastcrw.com/mcp/\u003cAPI_KEY\u003e. No key scopes found. A self-hosted engine needs no key unless `[auth] api_keys` is set.", "pricing": "freemium", "pricingNotes": "500 one-off credits with no card, plus up to 500 more from onboarding tasks. Hobby $13 a month for 5,000 credits, Standard $69 for 100,000, Growth $279 for 500,000, Scale $549 for 1,000,000 (less with yearly billing). One credit a page on any renderer, search 2. Failed pages refunded, top-ups never expire. Self-hosting is free.", "priceSummary": "$13 / mo", "where": "both", "x402": { "level": "no", "evidence": "No x402, MPP or L402 on the pricing page, the credit-costs and agent-onboarding docs, or the OpenAPI spec (checked 2026-11-06). A 402 on the hosted API means an exhausted credit balance (`errorCode: insufficient_credits`) with an upgrade URL header, not a payment requirement.", "endpoints": [] }, "toolCount": 11, "popularity": { "githubStars": 1200, "npmWeekly": 881, "pypiWeekly": null, "asOf": "2026-11-06" }, "docsUrl": "https://docs.fastcrw.com", "llmsTxt": "https://docs.fastcrw.com/llms.txt", "openapi": "https://github.com/fastcrw/crw/blob/main/docs/openapi.json", "registryName": "io.github.fastcrw/crw", "capabilities": [ "web.scrape", "web.crawl", "web.extract", "scraping.js-render", "scraping.proxies", "web.search", "web.fetch" ], "tags": [ "hosted", "local", "open-source", "self-hosted", "freemium", "no-card", "mcp", "remote-mcp", "mcp-registry", "llms-txt", "openapi", "rust", "eu" ], "lastRelease": "2026-11-04", "graded": true, "anchor": { "graded": true, "score": 64.3, "grade": "B", "agentReady": false, "rank": 478, "ranked": true, "rankOf": 1486, "categoryRank": 11, "methodology": "0.7", "run": "2026-10-01", "scores": { "ergonomics": 91, "maintenance": 93, "payments": 35, "reliability": 50, "schema": 91, "security": 40, "transparency": 60 }, "pending": [ "performance", "tasks" ], "assessment": { "confidence": "medium", "date": "2026-11-06" }, "negative": 0, "verdict": "Agents get 11 compact, annotated MCP tools with bounded output, a public OpenAPI spec, refunds on failed pages and a free self-hosted engine. The hosted service has no status history, publishes no SLA terms, names its operator only on the support page, and documents an MCP address with the API key in the URL path.", "bestFor": "Agents that want Firecrawl-style scrape, crawl, map, search and extract at lower per-page cost, or a small self-hosted engine.", "strengths": [ "11 MCP tools, each annotated readOnlyHint, destructiveHint and idempotentHint, with output capped near 15,000 characters and a resume offset", "Public OpenAPI 3.1 spec, llms.txt and an error page that says which codes to retry and which to stop on", "Failed and blocked pages are refunded, every renderer costs 1 credit, and top-up credits never expire", "Released 16 times in the 90 days to 6 November 2026, with five open issues on GitHub", "AGPL-3.0 engine runs self-hosted with no account, and MIT Python and TypeScript SDKs" ], "weaknesses": [ "The status page shows a live check and an uptime objective but no incident history", "The hosted MCP address fastcrw.com/mcp/\u003cAPI_KEY\u003e puts the key in the URL path as a documented option", "Terms and privacy policy name no legal entity, give no retention periods and name no DPA; only the support page names Nanofy OÜ", "No SLA terms beyond an Enterprise mention, no security.txt and no certification found", "No machine payment protocol, and hosted keys need a person to sign up in a browser" ], "agentNotes": [ "Connect to https://fastcrw.com/mcp with OAuth rather than the /mcp/\u003cAPI_KEY\u003e address, so the key stays out of URLs and logs", "Branch on `errorCode`. Stop on 402 insufficient_credits or llm_not_available, and on 429 wait for Retry-After before retrying", "Check `success` as well as the HTTP status, since an anti-bot wall answers 200 with `success: false` and `errorCode: anti_bot`", "Pass `startIndex` from a truncated result to read the rest of a long page instead of setting `maxLength: 0`", "Use native /v1 routes for new work; /firecrawl/v2 is for Firecrawl SDK migrations and keeps the older 429 for empty balances" ], "metrics": { "kind": "remote", "measured": false }, "reviewCount": 0, "avgRating": 0, "history": [ { "basis": "public evidence", "confidence": "medium", "grade": "B", "methodology": "0.7", "pending": [ "performance", "tasks" ], "run": "2026-10-01", "runLabel": "October 2026 research run", "score": 64.3 } ], "editorialScores": { "ergonomics": 91, "maintenance": 93, "payments": 35, "reliability": 50, "schema": 91, "security": 40, "transparency": 53 }, "provenanceScore": 67, "checked": { "categories": { "ergonomics": "2026-11-06", "maintenance": "2026-11-06", "payments": "2026-11-06", "reliability": "2026-11-06", "schema": "2026-11-06", "security": "2026-11-06", "transparency": "2026-11-06" }, "last": "2026-11-06", "oldest": "2026-11-06", "stale": false, "staleAfterDays": 90 } }, "connect": { "install": "npx -y crw-mcp@latest install", "http": "curl -X POST https://api.fastcrw.com/v1/scrape -H \"Authorization: Bearer YOUR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"url\":\"https://example.com\",\"formats\":[\"markdown\"]}'" }, "letme": { "capability": "https://letme.dev/web.scrape", "tool": "https://letme.dev/crw" }, "area": "web-data", "unitPrices": [ { "item": "Hobby plan", "unit": "month", "usd": 13, "note": "5,000 credits; $11 a month billed yearly" }, { "item": "Standard plan", "unit": "month", "usd": 69, "note": "100,000 credits; $59 a month billed yearly" }, { "item": "Hobby top-up", "unit": "1k-pages", "usd": 9, "note": "one credit scrapes one page on any renderer" }, { "item": "Scale top-up", "unit": "1k-pages", "usd": 0.6, "note": "$299 for 500,000 credits" } ], "provenance": { "legalEntity": "Nanofy OÜ", "domain": "fastcrw.com", "domainRegistered": "2026-03-05", "endpointOnVendorDomain": true, "terms": "https://fastcrw.com/terms", "privacy": "https://fastcrw.com/privacy", "statusPage": "https://fastcrw.com/status", "changelog": "https://github.com/fastcrw/crw/blob/main/CHANGELOG.md", "securityTxt": "none", "checked": "2026-11-06", "notes": [ "Nanofy OÜ (registry code 17365846, Sepapaja tn 6, Tallinn) is named as operator on fastcrw.com/support. The terms (last updated 4 April 2026) and the privacy policy (undated) name only fastCRW and 'the jurisdiction in which fastCRW operates'.", "The terms forbid crawling sites against their terms or robots.txt and put responsibility for targets on the customer. No clause forbids automated access to fastcrw.com itself.", "fastcrw.com/robots.txt allows all crawlers except account, API and login paths; docs.fastcrw.com/robots.txt allows all.", "RDAP gives fastcrw.com a registration date of 5 March 2026.", "fastcrw.com/.well-known/security.txt answers 404. SECURITY.md in the repository names security@fastcrw.com and GitHub private vulnerability reporting." ], "score": 67 }, "pageJsonUrl": "https://www.anchorterminal.com/tools/crw.json", "live": { "slug": "crw", "probe": { "target": "https://fastcrw.com/mcp", "method": "get", "lastAt": "2026-11-07T18:20:19.526009674Z", "lastOk": true, "lastStatus": 401, "lastMs": 156, "lastNote": "asks for credentials", "authRequired": true, "uptime24h": 100, "uptime30d": 100, "p50ms24h": 80, "p95ms24h": 153, "samples24h": 200, "samples30d": 200, "days": [ { "date": "2026-11-06", "probes": 13, "ok": 13 }, { "date": "2026-11-07", "probes": 187, "ok": 187 } ] }, "vendorStatus": { "page": "https://fastcrw.com/status", "indicator": "unknown", "summary": "no machine-readable status found", "checkedAt": "2026-11-07T04:47:35.720864858Z" }, "versions": [ { "registry": "github", "name": "fastcrw/crw", "version": "v0.42.1", "released": "2026-11-06", "seenAt": "2026-11-06T23:18:48.669809831Z" }, { "registry": "mcp-registry", "name": "io.github.fastcrw/crw", "version": "0.42.1", "seenAt": "2026-11-07T18:16:28.95801333Z" }, { "registry": "npm", "name": "crw-mcp", "version": "0.42.1", "seenAt": "2026-11-06T23:18:46.329470936Z" }, { "registry": "npm", "name": "crw-sdk", "version": "0.42.1", "seenAt": "2026-11-06T23:18:46.739900633Z" }, { "registry": "pypi", "name": "crw", "version": "0.42.1", "released": "2026-11-06", "seenAt": "2026-11-06T23:18:48.550796676Z" } ], "githubStars": 1230, "npmWeekly": 881, "pypiWeekly": 441, "securityTxt": { "url": "https://fastcrw.com/.well-known/security.txt", "state": "none", "checkedAt": "2026-11-07T15:59:50.596126033Z" }, "llmsTxt": { "url": "https://docs.fastcrw.com/llms.txt", "ok": true, "status": 200, "checkedAt": "2026-11-07T15:32:13.945191217Z" }, "pages": [ { "url": "https://raw.githubusercontent.com/fastcrw/crw/main/CHANGELOG.md", "kind": "changelog", "status": 200, "checkedAt": "2026-11-07T14:20:49.992321515Z", "changedAt": "0001-01-01T00:00:00Z", "fingerprint": "572aa4127341" }, { "url": "https://fastcrw.com/privacy", "kind": "privacy", "status": 200, "checkedAt": "2026-11-07T14:09:41.233044196Z", "changedAt": "0001-01-01T00:00:00Z", "fingerprint": "56adab90a2ac" }, { "url": "https://fastcrw.com/terms", "kind": "terms", "status": 200, "checkedAt": "2026-11-07T14:09:43.278176432Z", "changedAt": "0001-01-01T00:00:00Z", "fingerprint": "4f55b83dc108" } ], "updatedAt": "2026-11-07T18:20:19.526009674Z" } }, "answer": "fastCRW scores 64.3 (B) on agent readiness against Diffbot's 56.1 (C), and leads in 4 of 7 scored categories. Diffbot leads on reliability, payments \u0026 pricing and transparency \u0026 trust.", "b": { "slug": "diffbot", "name": "Diffbot", "vendor": "Diffbot Technologies Corp.", "vendorUrl": "https://www.diffbot.com", "kind": "http-api", "category": "web-scrapers", "summary": "Diffbot is a hosted set of web data APIs from Diffbot Technologies Corp. of Menlo Park, California. Its Extract API classifies a page and returns typed JSON, alongside Crawl, Web Search, Natural Language and Knowledge Graph APIs.", "url": "https://www.anchorterminal.com/tools/diffbot", "markdownUrl": "https://www.anchorterminal.com/tools/diffbot.md", "slimMarkdownUrl": "https://www.anchorterminal.com/tools/diffbot.min.md", "jsonUrl": "https://www.anchorterminal.com/api/v1/tools/diffbot.json", "dns": "diffbot.grade.anchorterminal.com", "repo": "https://github.com/diffbot/diffbot-python", "license": "Proprietary service under Diffbot's Terms of Use. The Python client, the TypeScript client and the agent skills are MIT", "transports": [ "http" ], "remoteUrl": "https://api.diffbot.com/v3", "packages": [ { "registry": "pypi", "name": "diffbot" }, { "registry": "npm", "name": "@diffbot/typescript" } ], "auth": "api-key", "authNotes": "Self-serve token from the dashboard at app.diffbot.com, sent as the `token` query parameter on Extract and Knowledge Graph calls. The official client sends it as a Bearer header to the Web Search endpoint. Tokens carry no scopes. Child tokens have the same rights as the primary token and can be created in the dashboard on the Plus plan and above, or through support. A changelog entry of 30 June 2025 is titled as adding labels and a disable switch for tokens in the dashboard.", "pricing": "freemium", "pricingNotes": "Free plan of 10,000 credits a month at 5 requests a minute, with no card, so an agent's owner can start without a contract. Startup is $299 a month for 250,000 credits with overage at $0.001 a credit, Plus $899 for 1,000,000 with overage at $0.0009 and access to Crawl, Enterprise by sales. One extracted page is 1 credit, or 2 through the data centre proxy. Whether a failed extraction uses a credit was not found (https://www.diffbot.com/pricing).", "priceSummary": "$1 / 1k pages", "where": "hosted", "x402": { "level": "no", "evidence": "No x402, MPP or L402 on the pricing page, the authentication and rate limit docs, the Extract OpenAPI file or the official clients (checked 2026-10-09).", "endpoints": [] }, "toolCount": null, "popularity": { "githubStars": null, "npmWeekly": 9, "pypiWeekly": null, "asOf": "2026-10-09" }, "docsUrl": "https://www.diffbot.com/docs/", "llmsTxt": "https://www.diffbot.com/llms.txt", "openapi": "https://www.diffbot.com/openapi/extract.json", "capabilities": [ "web.extract", "web.scrape", "web.crawl", "scraping.proxies", "web.search", "data.knowledge", "data.company" ], "tags": [ "hosted", "freemium", "free-tier", "no-card", "api-key", "openapi", "llms-txt", "agent-skills", "cli", "python", "typescript", "proxies", "knowledge-graph", "status-page", "closed-source" ], "lastRelease": "2026-10-29", "graded": true, "anchor": { "graded": true, "score": 56.1, "grade": "C", "agentReady": false, "rank": 840, "ranked": true, "rankOf": 1486, "categoryRank": 23, "methodology": "0.7", "run": "2026-10-01", "scores": { "ergonomics": 63, "maintenance": 77, "payments": 40, "reliability": 55, "schema": 83, "security": 21, "transparency": 68 }, "pending": [ "performance", "tasks" ], "assessment": { "confidence": "medium", "date": "2026-11-04" }, "negative": 0, "verdict": "Extract returns typed JSON for articles, products, discussions and other page types with no selectors to write, under a public OpenAPI 3.1 contract and a free plan that needs no card. The API token travels in the URL query string, tokens carry no scopes, and no security page, certification or security.txt was found.", "bestFor": "Agents that want typed article, product or discussion data from arbitrary pages without writing selectors, and teams that also want the Knowledge Graph.", "strengths": [ "OpenAPI 3.1.0 description of the Extract APIs (version 1.1.0, 13 operations) is published at `/openapi/extract.json` and the reference pages are generated from it", "Free plan of 10,000 credits a month with no card, and a public credit table. One extracted page costs 1 credit, or 2 through the data centre proxy", "Official Python and TypeScript clients and ten agent skills, all MIT. The Python client reached 3.1.0 on 29 October 2026 with tests on Python 3.9 to 3.13", "Dated changelog with an RSS feed going back to 2015, and a `/v3` path for the Extract APIs", "The privacy policy of 29 August 2025 names each service provider, among them AWS, Stripe, Intercom and Salesforce, and says all are in the United States" ], "weaknesses": [ "The token is sent as the `token` query parameter on Extract and Knowledge Graph calls. It is the only scheme in the OpenAPI file", "Child tokens have the same access rights as the primary token, and creating them needs the Plus plan ($899 a month) or a request to support", "No security page, certification, bug bounty or security.txt was found on the pages read. The privacy policy says some API calls rely on HTTP without encryption", "The status page is a Pingdom report with seven days of uptime per check and no incident notes. Analyze API shows 99.69 per cent for the week to 9 October 2026", "The docs give `pip install diffbot-python` and `pnpm add diffbot-typescript`, while the repositories now name the packages `diffbot` and `@diffbot/typescript`", "The terms grant Diffbot a perpetual, irrevocable licence to process submitted queries, cap liability at $10, and forbid software that scrapes or crawls the Site or Service. That clause matters before any probe is run" ], "agentNotes": [ "Call `GET https://api.diffbot.com/v3/analyze?url=\u003cencoded URL\u003e\u0026token=\u003ctoken\u003e` when the page type is unknown. Use `/v3/article` or `/v3/product` to force one schema", "Keep the token out of logs and shared URLs. It sits in the query string, so any proxy or log that records the request URL records the token", "Stay under 5 calls a minute on Free, 5 a second on Startup and 25 a second on Plus. On 429 wait at least one second and back off", "Check a 200 response for `errorCode`. The official client raises `ExtractionError` when a 200 reports a failed extraction", "Set `useProxy=default` only when a site blocks the fetch. A proxied page costs 2 credits against 1", "Install `diffbot` from PyPI, not `diffbot-python`, which stopped at 0.3.0" ], "metrics": { "kind": "remote", "measured": false }, "reviewCount": 0, "avgRating": 0, "history": [ { "basis": "public evidence", "confidence": "medium", "grade": "C", "methodology": "0.7", "pending": [ "performance", "tasks" ], "run": "2026-10-01", "runLabel": "October 2026 research run", "score": 56.1 } ], "editorialScores": { "ergonomics": 63, "maintenance": 77, "payments": 40, "reliability": 55, "schema": 83, "security": 21, "transparency": 49 }, "provenanceScore": 87, "checked": { "categories": { "ergonomics": "2026-11-04", "maintenance": "2026-11-04", "payments": "2026-11-04", "reliability": "2026-11-04", "schema": "2026-11-04", "security": "2026-11-04", "transparency": "2026-11-04" }, "last": "2026-11-04", "lastMoved": "2026-11-04", "oldest": "2026-11-04", "stale": false, "staleAfterDays": 90 } }, "connect": { "install": "python3 -m pip install diffbot", "http": "curl --request GET \\\n --url 'https://api.diffbot.com/v3/analyze?token=\u003cDIFFBOT_TOKEN\u003e\u0026url=https%3A%2F%2Fwww.diffbot.com%2Fdocs%2Fintroduction' \\\n --header 'accept: application/json'", "claudeCode": "/plugin marketplace add diffbot/diffbot-skills\n/plugin install diffbot@diffbot-skills" }, "letme": { "capability": "https://letme.dev/web.extract", "tool": "https://letme.dev/diffbot" }, "area": "web-data", "unitPrices": [ { "item": "Extracted page, Startup overage", "unit": "1k-pages", "usd": 1, "note": "1 credit a page at $0.001 a credit" }, { "item": "Extracted page through the data centre proxy, Startup overage", "unit": "1k-pages", "usd": 2, "note": "2 credits a page" }, { "item": "Extracted page, Plus overage", "unit": "1k-pages", "usd": 0.9, "note": "1 credit a page at $0.0009 a credit" }, { "item": "Startup plan", "unit": "month", "usd": 299, "note": "250,000 credits, 5 requests a second" }, { "item": "Plus plan", "unit": "month", "usd": 899, "note": "1,000,000 credits, 25 requests a second, Crawl" } ], "provenance": { "legalEntity": "Diffbot Technologies Corp.", "domain": "diffbot.com", "domainRegistered": "2004-08-14", "endpointOnVendorDomain": true, "terms": "https://www.diffbot.com/company/terms", "privacy": "https://www.diffbot.com/company/privacy", "statusPage": "https://status.diffbot.com", "changelog": "https://www.diffbot.com/changelog", "securityTxt": "none", "checked": "2026-10-09", "notes": [ "The Terms of Use name Diffbot Technologies Corp. as operator and cover both the Site and the Service, defined as the service that converts a web page into entities and facts. They are the only terms the site links, carry no date, and are the document that governs the API.", "The privacy policy was last updated on 29 August 2025, covers visitors, subscribers and people in the index, and gives the address 333 Ravenswood Ave, Menlo Park, CA 94025.", "www.diffbot.com/.well-known/security.txt returned 404. No security or trust page is linked from the home page or footer.", "The Extract APIs answer at api.diffbot.com, the Knowledge Graph at kg.diffbot.com, Natural Language at nl.diffbot.com and Web Search at llm.diffbot.com, per the docs and the official client's defaults.", "RDAP at Verisign gives a registration date of 2004-08-14 for diffbot.com.", "robots.txt on www.diffbot.com allows every path. status.diffbot.com disallows only Alexabot." ], "score": 87 }, "pageJsonUrl": "https://www.anchorterminal.com/tools/diffbot.json", "live": { "slug": "diffbot", "probe": { "target": "https://api.diffbot.com/v3", "method": "get", "lastAt": "2026-11-07T18:20:21.059828225Z", "lastOk": true, "lastStatus": 401, "lastMs": 1256, "lastNote": "asks for credentials", "authRequired": true, "uptime24h": 99.58, "uptime30d": 99.99, "p50ms24h": 447, "p95ms24h": 1134, "samples24h": 239, "samples30d": 7205, "days": [ { "date": "2026-10-09", "probes": 85, "ok": 85 }, { "date": "2026-10-10", "probes": 247, "ok": 247 }, { "date": "2026-10-11", "probes": 247, "ok": 247 }, { "date": "2026-10-12", "probes": 247, "ok": 247 }, { "date": "2026-10-13", "probes": 248, "ok": 248 }, { "date": "2026-10-14", "probes": 249, "ok": 249 }, { "date": "2026-10-15", "probes": 253, "ok": 253 }, { "date": "2026-10-16", "probes": 252, "ok": 252 }, { "date": "2026-10-17", "probes": 253, "ok": 253 }, { "date": "2026-10-18", "probes": 256, "ok": 256 }, { "date": "2026-10-19", "probes": 255, "ok": 255 }, { "date": "2026-10-20", "probes": 252, "ok": 252 }, { "date": "2026-10-21", "probes": 253, "ok": 253 }, { "date": "2026-10-22", "probes": 253, "ok": 253 }, { "date": "2026-10-23", "probes": 248, "ok": 248 }, { "date": "2026-10-24", "probes": 251, "ok": 251 }, { "date": "2026-10-25", "probes": 250, "ok": 250 }, { "date": "2026-10-26", "probes": 252, "ok": 252 }, { "date": "2026-10-27", "probes": 249, "ok": 249 }, { "date": "2026-10-28", "probes": 251, "ok": 251 }, { "date": "2026-10-29", "probes": 248, "ok": 248 }, { "date": "2026-10-30", "probes": 249, "ok": 249 }, { "date": "2026-10-31", "probes": 251, "ok": 251 }, { "date": "2026-11-01", "probes": 249, "ok": 249 }, { "date": "2026-11-02", "probes": 250, "ok": 250 }, { "date": "2026-11-03", "probes": 248, "ok": 248 }, { "date": "2026-11-04", "probes": 246, "ok": 246 }, { "date": "2026-11-05", "probes": 185, "ok": 185 }, { "date": "2026-11-06", "probes": 241, "ok": 241 }, { "date": "2026-11-07", "probes": 187, "ok": 186 } ] }, "vendorStatus": { "page": "https://status.diffbot.com", "indicator": "unknown", "summary": "no machine-readable status found", "checkedAt": "2026-11-07T04:47:38.761125171Z" }, "versions": [ { "registry": "github", "name": "diffbot/diffbot-python", "version": "v3.1.0", "released": "2026-10-29", "seenAt": "2026-11-06T23:21:21.943716388Z" }, { "registry": "npm", "name": "@diffbot/typescript", "version": "0.3.0", "seenAt": "2026-11-06T23:21:21.609607359Z" }, { "registry": "pypi", "name": "diffbot", "version": "3.1.0", "released": "2026-10-29", "seenAt": "2026-11-06T23:21:21.296993142Z" } ], "githubStars": 126, "npmWeekly": 208, "pypiWeekly": 56, "securityTxt": { "url": "https://diffbot.com/.well-known/security.txt", "state": "none", "checkedAt": "2026-11-07T16:01:16.162441258Z" }, "llmsTxt": { "url": "https://www.diffbot.com/llms.txt", "ok": true, "status": 200, "checkedAt": "2026-11-07T15:32:17.672259501Z" }, "domain": { "domain": "diffbot.com", "registered": "2004-08-14", "source": "https://rdap.verisign.com/com/v1/domain/diffbot.com", "checkedAt": "2026-11-01T14:13:22.626872253Z" }, "pages": [ { "url": "https://www.diffbot.com/changelog", "kind": "changelog", "status": 200, "checkedAt": "2026-11-07T14:27:53.356515934Z", "changedAt": "2026-10-30T05:54:43.227180558Z", "fingerprint": "aafbed2345a0" }, { "url": "https://www.diffbot.com/pricing", "kind": "pricing", "status": 200, "checkedAt": "2026-11-07T14:27:59.7529626Z", "changedAt": "0001-01-01T00:00:00Z", "fingerprint": "86884503041e" }, { "url": "https://www.diffbot.com/company/privacy", "kind": "privacy", "status": 200, "checkedAt": "2026-11-07T14:27:55.815662966Z", "changedAt": "2026-10-17T23:15:07.243669117Z", "fingerprint": "53b83d48cccd" }, { "url": "https://www.diffbot.com/company/terms", "kind": "terms", "status": 200, "checkedAt": "2026-11-07T14:27:58.141796415Z", "changedAt": "0001-01-01T00:00:00Z", "fingerprint": "a76686da1f94" } ], "github": { "repo": "diffbot/diffbot-python", "checkedAt": "2026-11-02T08:44:16.252933242Z", "stars": 126, "archived": false, "issuesEnabled": true, "pushedAt": "2026-10-29T23:08:37Z", "openIssues": 0, "openPRs": 0, "issuesOpened90d": 0, "issuesClosed90d": 0, "prsMerged90d": 0, "ci": { "branch": "main", "runs": 20, "success": 20, "failure": 0, "other": 0, "latest": "success", "latestAt": "2026-10-29T23:07:32Z" } }, "updatedAt": "2026-11-07T18:20:21.059828225Z" } }, "edges": [ { "by": 28, "category": "ergonomics", "leads": "crw", "name": "Agent ergonomics", "reasons": { "a": "11 compact MCP tools, two of them the data-source tools added in 0.41.0. Content is capped near 15,000 characters with `truncated: true` and a resume offset, and map returns at most 100 URLs by default (20 of 25). Formats, main-content filtering, include and exclude selectors, maxLength, startIndex, limit and crawl depth and page caps (20). Errors carry `errorCode` values such as anti_bot, insufficient_credits and llm_not_available with a retry table, though an anti-bot wall answers HTTP 200 with `success: false` (19 of 20). Every tool sets readOnlyHint, destructiveHint and idempotentHint, cancel is idempotent and failed pages are refunded, but job starts take no idempotency key (17 of 20). Only `url` is required, with official Python and TypeScript SDKs (15). 20 + 20 + 19 + 17 + 15 = 91.", "b": "Graded on the REST API, with the skills noted. `mode` limits extraction to one page type, `fields` adds optional fields and `discussion=false` drops comments, but an article response carries both `html` and `text` with no way found to drop either. The skills return Markdown by default (14 of 25). A single-URL call has no pagination. Web Search takes `size` and `maxTokens` and DQL takes `size` per the client. Crawl result paging was not read (10 of 20). Errors are JSON with `errorCode` and `error` and six docs pages. A failed extraction can arrive in a 200, and the 429 text says the site received too many requests when the cause is the caller's plan limit (12 of 20). Extract calls are GET reads and safe to retry. Whether a failed call uses a credit was not found, and there is no idempotency key for creating crawl jobs or custom APIs (12 of 20). Only `url` and `token` are required, with official Python and TypeScript clients and a CLI (15). Docs per the 9 October check, not re-read on 4 November. The Python client 3.1.0 (29 October 2026) adds a `snippet_length` option that maps to the Web Search API's `snippetLength`. 14 + 10 + 12 + 12 + 15 = 63." }, "scores": { "a": 91, "b": 63 } }, { "by": 19, "category": "security", "leads": "crw", "name": "Security \u0026 auth", "reasons": { "a": "The hosted MCP server publishes RFC 9728 metadata naming its own authorisation server, with PKCE S256, dynamic client registration and one scope, `mcp:tools`, and the API takes revocable Bearer keys from the dashboard. No key scopes or second scope found (22). The authentication page documents fastcrw.com/mcp/\u003cAPI_KEY\u003e as a way in, a key in the URL path, so less 10 (12 of 30). Tools mostly read, and data-source calls check approvals, but there is no read-only key or confirmation step for monitors (8 of 20). Managed LLM extraction, summaries and answers wrap scraped text in a nonce fence (crates/crw-extract/src/untrusted.rs), and the hardening guide treats pages as untrusted, but the MCP docs give agents no guidance on scraped content (7 of 15). Each response reports `creditsUsed`; no per-call log in the docs (5 of 15). SECURITY.md names security@fastcrw.com and GitHub private reporting with a 48-hour acknowledgement, and CI runs CodeQL and OpenSSF Scorecard. security.txt answers 404, and no bounty or certification found (8 of 20). 12 + 8 + 7 + 5 + 8 = 40.", "b": "One token per account, sent in the URL query string as the documented and only scheme in the OpenAPI file. Tokens are revocable in the dashboard per a June 2025 changelog title, and child tokens exist on Plus and above (20 of 30, less 10 for the secret in the URL, so 10). Child tokens have the same rights as the primary one and there is no read-only mode. The skills limit the agent to the `db` binary through `allowed-tools` (5 of 20). The API returns untrusted web content and no guidance on prompt injection was found (0 of 15). The token details page shows usage activity and an Account API reports usage. No per-call log was found (6 of 15). No security.txt, security page, certification or bug bounty was found on the pages read, and the privacy policy says some API calls rely on HTTP without encryption (0 of 20). Site pages per the 9 October check, not re-read on 4 November. OSV lists no advisory for `diffbot` on PyPI or `@diffbot/typescript` on npm on 4 November. 10 + 5 + 0 + 6 + 0 = 21." }, "scores": { "a": 40, "b": 21 } }, { "by": 16, "category": "maintenance", "leads": "crw", "name": "Maintenance \u0026 community", "reasons": { "a": "v0.41.0 on 4 November 2026, matching the registry and npm (30). 16 dated releases in the 90 days from 8 August, from 0.37.0 to 0.41.0 (20). Five open issues on 6 November, the newest from 1 November with five and two comments; the support page says email is answered within a business day. Reply authors weren't read (18 of 25). In the official MCP registry as io.github.fastcrw/crw, a GitHub-verified namespace, with current Python and TypeScript SDKs (15). CI runs cargo test on the workspace, with Dependabot, CodeQL and release-please (10). 30 + 20 + 18 + 15 + 10 = 93.", "b": "Every dated source of the vendor's own was looked for in its GitHub repositories, since its site wasn't re-read. The official Python client, `diffbot` in the listing's packages, was tagged v3.1.0 on 29 October 2026, adding a Web Search `snippet_length` option, and the agent skills were tagged v1.2.0 the same day with rewritten skill content, 6 days before this check (30; was 10 on the API changelog's 15 June 2026 entry, which is per the 9 October check, not re-read on 4 November). In the 90 days from 6 August, the skills were tagged v1.1.0 (12 August, five new skills), v1.1.1 (13 August, a plugin marketplace manifest, counted half) and v1.2.0, and the Python client v3.0.0 (28 September) and v3.1.0, counted half as SDK tags, 3.5 in all (20). A public changelog with RSS and a support address exist. GitHub issues were not read (8 of 15 for a closed service). Python and TypeScript clients exist, but the docs named the packages `diffbot-python` and `diffbot-typescript` on 9 October while the repositories publish `diffbot` and `@diffbot/typescript`, and the TypeScript repository has no tag and no commit since 17 August 2026 (12 of 15). The Python repository tests on Python 3.9 to 3.13. We did not see CI results (7 of 10). 30 + 20 + 8 + 12 + 7 = 77." }, "scores": { "a": 93, "b": 77 } }, { "by": 8, "category": "schema", "leads": "crw", "name": "Schema \u0026 documentation", "reasons": { "a": "OpenAPI 3.1 for 11 paths at version 0.41.0 in docs/openapi.json, checked against the server in CI, and typed JSON Schema inputs on every MCP tool (25). llms.txt (86 lines), llms-full.txt and Markdown sources for every docs page (10). Tool descriptions state purpose and how to chain them (crawl returns a job id to poll, map returns URLs only, extract needs an LLM), and the docs have a choose-an-endpoint page, but few descriptions say when not to use a tool (14 of 20). Inputs are typed with required fields and a renderer enum, though crawl and extract take a free-form JSON Schema object (12 of 15). Examples on every page and an errors page with each HTTP status, errorCode and next step (15). Semver releases with a dated CHANGELOG that flags breaking changes (15). 25 + 10 + 14 + 12 + 15 + 15 = 91.", "b": "The Extract APIs have a public OpenAPI 3.1.0 file, version 1.1.0, with 13 operations, and the reference pages are generated from it. We read that file and one rendered page. The descriptions for Crawl, Knowledge Graph and Web Search were not read (25). llms.txt exists but is a short index with no Markdown twins of the docs pages found (5 of 10). The Analyze API page has sections on when to use it and when not to (17 of 20). Parameters are typed, `url` is the one required field and the file has 15 enums, but `fields` is a comma-separated string and the file defines no shared response schemas (10 of 15). The file carries 24 examples. It documents only 200 and 500, with separate docs pages for 401, 404, 429 and 457 (11 of 15). The path is versioned `/v3` and a dated changelog with RSS runs from 2015 to 15 June 2026 (15). All per the 9 October check, not re-read on 4 November (terms bar automated access to the site). 25 + 5 + 17 + 10 + 11 + 15 = 83." }, "scores": { "a": 91, "b": 83 } }, { "by": 8, "category": "transparency", "leads": "diffbot", "name": "Transparency \u0026 trust", "reasons": { "a": "The engine and MCP server are AGPL-3.0 and the SDKs MIT, while the hosted control plane is closed under terms that name no legal entity (25 of 30). The terms say crawled data is handled transiently with zero data retention for approved accounts, but the privacy policy is undated, names no controller, gives no retention periods and mentions no DPA (10 of 30). The CHANGELOG flags breaking changes, deprecated v2 extract routes return a warning on every response, and the October 2026 402 change is dated in the docs, with no standing notice policy (10 of 20). Local data-source telemetry is disclosed with an opt-out. The privacy policy names Stripe, Google Analytics, Google Ads and an X pixel, but infrastructure and search providers are unnamed and no data location is given (8 of 20). 25 + 10 + 10 + 8 = 53.", "b": "A closed service under undated Terms of Use that cover the Site and the Service, cap liability at $10 and grant Diffbot a perpetual licence to process submitted queries. The clients and skills are MIT (13 of 30). The privacy policy of 29 August 2025 says query history and API calls are stored against the API key and kept no longer than necessary, with no retention period given. The GDPR and data rights page was not read (14 of 30). There is no deprecation policy. The changelog carries dated notices, such as a May 2024 entry announcing a field removal for 30 August 2024, and the terms give 14 days' notice of fee changes (8 of 20). The privacy policy names its service providers one by one, among them AWS, Stripe, Intercom, Salesforce, Slack and Amplitude, and says all are in the United States (14 of 20). All per the 9 October check, not re-read on 4 November. 13 + 14 + 8 + 14 = 49." }, "scores": { "a": 60, "b": 68 } }, { "by": 5, "category": "reliability", "leads": "diffbot", "name": "Reliability", "reasons": { "a": "Graded on the hosted lines, for the hosted API and MCP server an agent would call; the self-hosted engine would read higher on the local lines. fastcrw.com/status is a first-party page with a live health check and a 99.9% monthly uptime objective, but no components with history (10 of 20). It lists no incidents and keeps no history we could read (5). Concurrency per plan is published, 3 on Free to 150 on Scale, while the request-rate guard above it is deliberately unpublished (12 of 15). The rate-limits page documents Retry-After and X-RateLimit headers, gives exponential backoff code, and separates 429 (retry) from 402 (stop). No idempotency keys for crawl, extract or monitor starts (13 of 15). Pricing mentions an SLA only as 'Premier + SLA' on Enterprise, with no terms, and the terms promise no uptime (0). The hosted service is sold on paid plans with no beta label, though the engine is at 0.41.0 (10). 10 + 5 + 12 + 13 + 0 + 10 = 50.", "b": "Graded as a hosted API. status.diffbot.com is a Pingdom public report with 13 checks, among them Analyze API, Article API, Crawlbot and Websearch, with daily uptime for seven days and no incident notes (15 of 20). No 90-day record can be read. In the week to 9 October 2026 Analyze API shows 99.69 per cent, with 2, 11 and 17 minutes down on 4, 5 and 6 October, and Article API 5, 9 and 2 minutes on 5 to 7 October (5 of 30, the line for no readable history). Rate limits are published per plan, 5 calls a minute on Free, 5 a second on Startup and 25 a second on Plus (15). The 429 page says to pause one second and back off. No `Retry-After` header is documented, though the official client reads one (10 of 15). No SLA was found (0). The Extract APIs are at `/v3` with no beta label (10). All per the 9 October check, not re-read on 4 November: Diffbot's terms bar software that scrapes or crawls the Site or Service, so its site and status report weren't fetched. 15 + 5 + 15 + 10 + 0 + 10 = 55." }, "scores": { "a": 50, "b": 55 } } ], "facts": [ { "a": "HTTP API", "b": "HTTP API", "name": "Kind" }, { "a": "Nanofy OÜ", "b": "Diffbot Technologies Corp.", "name": "Vendor" }, { "a": "https://fastcrw.com/mcp", "b": "https://api.diffbot.com/v3", "name": "Hosted endpoint" }, { "a": "stdio, Streamable HTTP", "b": "HTTP", "name": "Transports" }, { "a": "OAuth or key", "b": "API key", "name": "Auth" }, { "a": "Freemium", "b": "Freemium", "name": "Pricing" }, { "a": "no", "b": "no", "name": "x402" }, { "a": "AGPL-3.0", "b": "Proprietary service under Diffbot's Terms of Use. The Python client, the TypeScript client and the agent skills are MIT", "name": "Licence" }, { "a": "11", "b": "none", "name": "Tools exposed" }, { "a": "no", "b": "no", "name": "Read-only variant documented" }, { "a": "yes", "b": "yes", "name": "llms.txt" }, { "a": "io.github.fastcrw/crw", "b": "not listed", "name": "MCP registry" }, { "a": "2026-11-04", "b": "2026-10-29", "name": "Last release" }, { "a": "2026-04-04", "b": "no date given", "name": "Terms last updated" }, { "a": "no date given", "b": "2025-08-29", "name": "Privacy policy last updated" }, { "a": "not found in the text", "b": "not found in the text", "name": "Customer content may train models" }, { "a": "yes", "b": "yes", "name": "Terms restrict automated access" }, { "a": "yes", "b": "not found in the text", "name": "Terms restrict benchmarking" }, { "a": "not found in the text", "b": "not found in the text", "name": "Terms or service can change without notice" }, { "a": "not found in the text", "b": "yes", "name": "Arbitration or class-action waiver" }, { "a": "1.2k stars, 881 npm/wk", "b": "9 npm/wk", "name": "Popularity" } ], "faq": [ { "answer": "fastCRW scores 64.3 (B) on agent readiness against Diffbot's 56.1 (C), and leads in 4 of 7 scored categories. Diffbot leads on reliability, payments \u0026 pricing and transparency \u0026 trust.", "question": "Which is better for AI agents, fastCRW or Diffbot?" }, { "answer": "fastCRW takes an API key or an OAuth sign-in. Diffbot needs an API key.", "question": "Do fastCRW and Diffbot need an API key?" }, { "answer": "Yes. fastCRW has a hosted endpoint at https://fastcrw.com/mcp and Diffbot at https://api.diffbot.com/v3.", "question": "Can an agent call fastCRW and Diffbot without installing anything?" }, { "answer": "fastCRW is open source (AGPL-3.0). No open-source release is listed for Diffbot.", "question": "Are fastCRW and Diffbot open source?" } ], "goodFor": [ { "aheadOn": [ "Schema \u0026 documentation, 91 against 83", "Agent ergonomics, 91 against 63", "Security \u0026 auth, 40 against 21", "Maintenance \u0026 community, 93 against 77" ], "also": [ "Runs on your own machine", "Open source" ], "goodFor": "Agents that want Firecrawl-style scrape, crawl, map, search and extract at lower per-page cost, or a small self-hosted engine.", "slug": "crw", "watchFor": "The status page shows a live check and an uptime objective but no incident history" }, { "aheadOn": [ "Reliability, 55 against 50", "Payments \u0026 pricing, 40 against 35", "Transparency \u0026 trust, 68 against 60" ], "also": null, "goodFor": "Agents that want typed article, product or discussion data from arbitrary pages without writing selectors, and teams that also want the Knowledge Graph.", "slug": "diffbot", "watchFor": "The token is sent as the `token` query parameter on Extract and Knowledge Graph calls. It is the only scheme in the OpenAPI file" } ], "history": { "crw": null, "diffbot": [ { "causes": [ "product" ], "date": "2026-11-04", "delta": 1.8, "from": 54.3, "fromGrade": "C", "to": 56.1, "toGrade": "C" } ] }, "job": { "capability": "web.scrape", "name": "Scraping" }, "others": [ { "json": "https://www.anchorterminal.com/compare/bright-data-vs-crw.json", "title": "Bright Data vs fastCRW", "url": "https://www.anchorterminal.com/compare/bright-data-vs-crw" }, { "json": "https://www.anchorterminal.com/compare/bright-data-vs-diffbot.json", "title": "Bright Data vs Diffbot", "url": "https://www.anchorterminal.com/compare/bright-data-vs-diffbot" }, { "json": "https://www.anchorterminal.com/compare/browseract-vs-crw.json", "title": "BrowserAct vs fastCRW", "url": "https://www.anchorterminal.com/compare/browseract-vs-crw" }, { "json": "https://www.anchorterminal.com/compare/browseract-vs-diffbot.json", "title": "BrowserAct vs Diffbot", "url": "https://www.anchorterminal.com/compare/browseract-vs-diffbot" }, { "json": "https://www.anchorterminal.com/compare/context-dev-vs-crw.json", "title": "Context.dev vs fastCRW", "url": "https://www.anchorterminal.com/compare/context-dev-vs-crw" }, { "json": "https://www.anchorterminal.com/compare/context-dev-vs-diffbot.json", "title": "Context.dev vs Diffbot", "url": "https://www.anchorterminal.com/compare/context-dev-vs-diffbot" }, { "json": "https://www.anchorterminal.com/compare/crawl4ai-vs-crw.json", "title": "Crawl4AI vs fastCRW", "url": "https://www.anchorterminal.com/compare/crawl4ai-vs-crw" }, { "json": "https://www.anchorterminal.com/compare/crawl4ai-vs-diffbot.json", "title": "Crawl4AI vs Diffbot", "url": "https://www.anchorterminal.com/compare/crawl4ai-vs-diffbot" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-decodo.json", "title": "fastCRW vs Decodo Web Data API", "url": "https://www.anchorterminal.com/compare/crw-vs-decodo" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-jevcrawl.json", "title": "fastCRW vs jevcrawl", "url": "https://www.anchorterminal.com/compare/crw-vs-jevcrawl" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-jina-reader.json", "title": "fastCRW vs Jina Reader", "url": "https://www.anchorterminal.com/compare/crw-vs-jina-reader" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-microlink.json", "title": "fastCRW vs Microlink", "url": "https://www.anchorterminal.com/compare/crw-vs-microlink" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-olostep.json", "title": "fastCRW vs Olostep", "url": "https://www.anchorterminal.com/compare/crw-vs-olostep" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-outcrawl.json", "title": "fastCRW vs Outcrawl", "url": "https://www.anchorterminal.com/compare/crw-vs-outcrawl" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-oxylabs-web-scraper.json", "title": "fastCRW vs Oxylabs Web Scraper API", "url": "https://www.anchorterminal.com/compare/crw-vs-oxylabs-web-scraper" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrape-do.json", "title": "fastCRW vs Scrape.do", "url": "https://www.anchorterminal.com/compare/crw-vs-scrape-do" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapegraphai.json", "title": "fastCRW vs ScrapeGraphAI", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapegraphai" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapeless.json", "title": "fastCRW vs Scrapeless", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapeless" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapeops.json", "title": "fastCRW vs ScrapeOps Proxy API Aggregator", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapeops" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scraperapi.json", "title": "fastCRW vs ScraperAPI", "url": "https://www.anchorterminal.com/compare/crw-vs-scraperapi" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapewise.json", "title": "fastCRW vs Scrapewise", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapewise" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapfly.json", "title": "fastCRW vs Scrapfly", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapfly" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapingant.json", "title": "fastCRW vs ScrapingAnt", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapingant" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapingbee.json", "title": "fastCRW vs ScrapingBee", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapingbee" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapingdog.json", "title": "fastCRW vs Scrapingdog", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapingdog" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-spider-cloud.json", "title": "fastCRW vs Spider", "url": "https://www.anchorterminal.com/compare/crw-vs-spider-cloud" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-webscraping-ai.json", "title": "fastCRW vs WebScraping.AI", "url": "https://www.anchorterminal.com/compare/crw-vs-webscraping-ai" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-zenrows.json", "title": "fastCRW vs ZenRows", "url": "https://www.anchorterminal.com/compare/crw-vs-zenrows" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-zyte.json", "title": "fastCRW vs Zyte API", "url": "https://www.anchorterminal.com/compare/crw-vs-zyte" }, { "json": "https://www.anchorterminal.com/compare/decodo-vs-diffbot.json", "title": "Decodo Web Data API vs Diffbot", "url": "https://www.anchorterminal.com/compare/decodo-vs-diffbot" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-jina-reader.json", "title": "Diffbot vs Jina Reader", "url": "https://www.anchorterminal.com/compare/diffbot-vs-jina-reader" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrape-do.json", "title": "Diffbot vs Scrape.do", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrape-do" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapeless.json", "title": "Diffbot vs Scrapeless", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapeless" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-firecrawl-mcp.json", "title": "fastCRW vs Firecrawl MCP", "url": "https://www.anchorterminal.com/compare/crw-vs-firecrawl-mcp" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-qovia.json", "title": "fastCRW vs Qovia", "url": "https://www.anchorterminal.com/compare/crw-vs-qovia" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-scrapeatlas.json", "title": "fastCRW vs ScrapeAtlas", "url": "https://www.anchorterminal.com/compare/crw-vs-scrapeatlas" }, { "json": "https://www.anchorterminal.com/compare/crw-vs-soldfetch.json", "title": "fastCRW vs SoldFetch", "url": "https://www.anchorterminal.com/compare/crw-vs-soldfetch" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-firecrawl-mcp.json", "title": "Diffbot vs Firecrawl MCP", "url": "https://www.anchorterminal.com/compare/diffbot-vs-firecrawl-mcp" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-jevcrawl.json", "title": "Diffbot vs jevcrawl", "url": "https://www.anchorterminal.com/compare/diffbot-vs-jevcrawl" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-microlink.json", "title": "Diffbot vs Microlink", "url": "https://www.anchorterminal.com/compare/diffbot-vs-microlink" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-olostep.json", "title": "Diffbot vs Olostep", "url": "https://www.anchorterminal.com/compare/diffbot-vs-olostep" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-outcrawl.json", "title": "Diffbot vs Outcrawl", "url": "https://www.anchorterminal.com/compare/diffbot-vs-outcrawl" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-oxylabs-web-scraper.json", "title": "Diffbot vs Oxylabs Web Scraper API", "url": "https://www.anchorterminal.com/compare/diffbot-vs-oxylabs-web-scraper" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-qovia.json", "title": "Diffbot vs Qovia", "url": "https://www.anchorterminal.com/compare/diffbot-vs-qovia" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapeatlas.json", "title": "Diffbot vs ScrapeAtlas", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapeatlas" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapegraphai.json", "title": "Diffbot vs ScrapeGraphAI", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapegraphai" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapeops.json", "title": "Diffbot vs ScrapeOps Proxy API Aggregator", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapeops" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scraperapi.json", "title": "Diffbot vs ScraperAPI", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scraperapi" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapewise.json", "title": "Diffbot vs Scrapewise", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapewise" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapfly.json", "title": "Diffbot vs Scrapfly", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapfly" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapingant.json", "title": "Diffbot vs ScrapingAnt", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapingant" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapingbee.json", "title": "Diffbot vs ScrapingBee", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapingbee" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-scrapingdog.json", "title": "Diffbot vs Scrapingdog", "url": "https://www.anchorterminal.com/compare/diffbot-vs-scrapingdog" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-soldfetch.json", "title": "Diffbot vs SoldFetch", "url": "https://www.anchorterminal.com/compare/diffbot-vs-soldfetch" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-spider-cloud.json", "title": "Diffbot vs Spider", "url": "https://www.anchorterminal.com/compare/diffbot-vs-spider-cloud" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-webscraping-ai.json", "title": "Diffbot vs WebScraping.AI", "url": "https://www.anchorterminal.com/compare/diffbot-vs-webscraping-ai" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-zenrows.json", "title": "Diffbot vs ZenRows", "url": "https://www.anchorterminal.com/compare/diffbot-vs-zenrows" }, { "json": "https://www.anchorterminal.com/compare/diffbot-vs-zyte.json", "title": "Diffbot vs Zyte API", "url": "https://www.anchorterminal.com/compare/diffbot-vs-zyte" }, { "json": "https://www.anchorterminal.com/compare/apify-mcp-vs-crw.json", "title": "Apify MCP Server vs fastCRW", "url": "https://www.anchorterminal.com/compare/apify-mcp-vs-crw" }, { "json": "https://www.anchorterminal.com/compare/apify-mcp-vs-diffbot.json", "title": "Apify MCP Server vs Diffbot", "url": "https://www.anchorterminal.com/compare/apify-mcp-vs-diffbot" } ], "panel": null, "scores": [ { "by": 5, "crw": 50, "diffbot": 55, "edge": "diffbot", "key": "reliability", "name": "Reliability", "weight": 16 }, { "key": "performance", "name": "Performance", "pending": true, "weight": 10 }, { "by": 8, "crw": 91, "diffbot": 83, "edge": "crw", "key": "schema", "name": "Schema \u0026 documentation", "weight": 13 }, { "by": 28, "crw": 91, "diffbot": 63, "edge": "crw", "key": "ergonomics", "name": "Agent ergonomics", "weight": 13 }, { "by": 19, "crw": 40, "diffbot": 21, "edge": "crw", "key": "security", "name": "Security \u0026 auth", "weight": 14 }, { "by": 5, "crw": 35, "diffbot": 40, "edge": "diffbot", "key": "payments", "name": "Payments \u0026 pricing", "weight": 10 }, { "key": "tasks", "name": "Task success", "pending": true, "weight": 10 }, { "by": 16, "crw": 93, "diffbot": 77, "edge": "crw", "key": "maintenance", "name": "Maintenance \u0026 community", "weight": 7 }, { "by": 8, "crw": 60, "diffbot": 68, "edge": "diffbot", "key": "transparency", "name": "Transparency \u0026 trust", "weight": 7 } ], "summary": "fastCRW scores 64.3 (B) on agent readiness against Diffbot's 56.1 (C), and leads in 4 of 7 scored categories. Diffbot leads on reliability, payments \u0026 pricing and transparency \u0026 trust. Both do scraping.", "tasks": null, "tutorials": [], "verdicts": { "crw": "Agents get 11 compact, annotated MCP tools with bounded output, a public OpenAPI spec, refunds on failed pages and a free self-hosted engine. The hosted service has no status history, publishes no SLA terms, names its operator only on the support page, and documents an MCP address with the API key in the URL path.", "diffbot": "Extract returns typed JSON for articles, products, discussions and other page types with no selectors to write, under a public OpenAPI 3.1 contract and a free plan that needs no card. The API token travels in the URL query string, tokens carry no scopes, and no security page, certification or security.txt was found." } }, "kind": "anchor.page", "links": { "api": "https://www.anchorterminal.com/api/v1/index.json", "html": "https://www.anchorterminal.com/compare/crw-vs-diffbot", "json": "https://www.anchorterminal.com/compare/crw-vs-diffbot.json", "llms": "https://www.anchorterminal.com/llms.txt", "markdown": "https://www.anchorterminal.com/compare/crw-vs-diffbot.md", "mcp": "https://www.anchorterminal.com/mcp", "slim": "https://www.anchorterminal.com/compare/crw-vs-diffbot.min.md" }, "markdown": "fastCRW scores 64.3 (B) on agent readiness against Diffbot's 56.1 (C), and leads in 4 of 7 scored categories. Diffbot leads on reliability, payments \u0026 pricing and transparency \u0026 trust. Both do scraping.\n\n- fastCRW: grade B, 64.3/100, rank #478 of 1486. Markdown https://www.anchorterminal.com/tools/crw.md · JSON https://www.anchorterminal.com/api/v1/tools/crw.json\n- Diffbot: grade C, 56.1/100, rank #840 of 1486. Markdown https://www.anchorterminal.com/tools/diffbot.md · JSON https://www.anchorterminal.com/api/v1/tools/diffbot.json\n- Best web scraping and crawling APIs for AI agents: https://www.anchorterminal.com/best/web-scrapers/index.md\n- All 501 scrapers comparisons: https://www.anchorterminal.com/compare/web-scrapers/index.md\n\n## Which one, for what\n\n### fastCRW (B)\n\nGood for: Agents that want Firecrawl-style scrape, crawl, map, search and extract at lower per-page cost, or a small self-hosted engine.\n\nAhead on:\n- Schema \u0026 documentation, 91 against 83\n- Agent ergonomics, 91 against 63\n- Security \u0026 auth, 40 against 21\n- Maintenance \u0026 community, 93 against 77\n\nAlso in its favour:\n- Runs on your own machine\n- Open source\n\nWatch for: The status page shows a live check and an uptime objective but no incident history\n\n### Diffbot (C)\n\nGood for: Agents that want typed article, product or discussion data from arbitrary pages without writing selectors, and teams that also want the Knowledge Graph.\n\nAhead on:\n- Reliability, 55 against 50\n- Payments \u0026 pricing, 40 against 35\n- Transparency \u0026 trust, 68 against 60\n\nWatch for: The token is sent as the `token` query parameter on Extract and Knowledge Graph calls. It is the only scheme in the OpenAPI file\n\n\n## Why the edge\n\n### Agent ergonomics: fastCRW +28\n\n- fastCRW (91): 11 compact MCP tools, two of them the data-source tools added in 0.41.0. Content is capped near 15,000 characters with `truncated: true` and a resume offset, and map returns at most 100 URLs by default (20 of 25). Formats, main-content filtering, include and exclude selectors, maxLength, startIndex, limit and crawl depth and page caps (20). Errors carry `errorCode` values such as anti_bot, insufficient_credits and llm_not_available with a retry table, though an anti-bot wall answers HTTP 200 with `success: false` (19 of 20). Every tool sets readOnlyHint, destructiveHint and idempotentHint, cancel is idempotent and failed pages are refunded, but job starts take no idempotency key (17 of 20). Only `url` is required, with official Python and TypeScript SDKs (15). 20 + 20 + 19 + 17 + 15 = 91.\n- Diffbot (63): Graded on the REST API, with the skills noted. `mode` limits extraction to one page type, `fields` adds optional fields and `discussion=false` drops comments, but an article response carries both `html` and `text` with no way found to drop either. The skills return Markdown by default (14 of 25). A single-URL call has no pagination. Web Search takes `size` and `maxTokens` and DQL takes `size` per the client. Crawl result paging was not read (10 of 20). Errors are JSON with `errorCode` and `error` and six docs pages. A failed extraction can arrive in a 200, and the 429 text says the site received too many requests when the cause is the caller's plan limit (12 of 20). Extract calls are GET reads and safe to retry. Whether a failed call uses a credit was not found, and there is no idempotency key for creating crawl jobs or custom APIs (12 of 20). Only `url` and `token` are required, with official Python and TypeScript clients and a CLI (15). Docs per the 9 October check, not re-read on 4 November. The Python client 3.1.0 (29 October 2026) adds a `snippet_length` option that maps to the Web Search API's `snippetLength`. 14 + 10 + 12 + 12 + 15 = 63.\n\n### Security \u0026 auth: fastCRW +19\n\n- fastCRW (40): The hosted MCP server publishes RFC 9728 metadata naming its own authorisation server, with PKCE S256, dynamic client registration and one scope, `mcp:tools`, and the API takes revocable Bearer keys from the dashboard. No key scopes or second scope found (22). The authentication page documents fastcrw.com/mcp/\u003cAPI_KEY\u003e as a way in, a key in the URL path, so less 10 (12 of 30). Tools mostly read, and data-source calls check approvals, but there is no read-only key or confirmation step for monitors (8 of 20). Managed LLM extraction, summaries and answers wrap scraped text in a nonce fence (crates/crw-extract/src/untrusted.rs), and the hardening guide treats pages as untrusted, but the MCP docs give agents no guidance on scraped content (7 of 15). Each response reports `creditsUsed`; no per-call log in the docs (5 of 15). SECURITY.md names security@fastcrw.com and GitHub private reporting with a 48-hour acknowledgement, and CI runs CodeQL and OpenSSF Scorecard. security.txt answers 404, and no bounty or certification found (8 of 20). 12 + 8 + 7 + 5 + 8 = 40.\n- Diffbot (21): One token per account, sent in the URL query string as the documented and only scheme in the OpenAPI file. Tokens are revocable in the dashboard per a June 2025 changelog title, and child tokens exist on Plus and above (20 of 30, less 10 for the secret in the URL, so 10). Child tokens have the same rights as the primary one and there is no read-only mode. The skills limit the agent to the `db` binary through `allowed-tools` (5 of 20). The API returns untrusted web content and no guidance on prompt injection was found (0 of 15). The token details page shows usage activity and an Account API reports usage. No per-call log was found (6 of 15). No security.txt, security page, certification or bug bounty was found on the pages read, and the privacy policy says some API calls rely on HTTP without encryption (0 of 20). Site pages per the 9 October check, not re-read on 4 November. OSV lists no advisory for `diffbot` on PyPI or `@diffbot/typescript` on npm on 4 November. 10 + 5 + 0 + 6 + 0 = 21.\n\n### Maintenance \u0026 community: fastCRW +16\n\n- fastCRW (93): v0.41.0 on 4 November 2026, matching the registry and npm (30). 16 dated releases in the 90 days from 8 August, from 0.37.0 to 0.41.0 (20). Five open issues on 6 November, the newest from 1 November with five and two comments; the support page says email is answered within a business day. Reply authors weren't read (18 of 25). In the official MCP registry as io.github.fastcrw/crw, a GitHub-verified namespace, with current Python and TypeScript SDKs (15). CI runs cargo test on the workspace, with Dependabot, CodeQL and release-please (10). 30 + 20 + 18 + 15 + 10 = 93.\n- Diffbot (77): Every dated source of the vendor's own was looked for in its GitHub repositories, since its site wasn't re-read. The official Python client, `diffbot` in the listing's packages, was tagged v3.1.0 on 29 October 2026, adding a Web Search `snippet_length` option, and the agent skills were tagged v1.2.0 the same day with rewritten skill content, 6 days before this check (30; was 10 on the API changelog's 15 June 2026 entry, which is per the 9 October check, not re-read on 4 November). In the 90 days from 6 August, the skills were tagged v1.1.0 (12 August, five new skills), v1.1.1 (13 August, a plugin marketplace manifest, counted half) and v1.2.0, and the Python client v3.0.0 (28 September) and v3.1.0, counted half as SDK tags, 3.5 in all (20). A public changelog with RSS and a support address exist. GitHub issues were not read (8 of 15 for a closed service). Python and TypeScript clients exist, but the docs named the packages `diffbot-python` and `diffbot-typescript` on 9 October while the repositories publish `diffbot` and `@diffbot/typescript`, and the TypeScript repository has no tag and no commit since 17 August 2026 (12 of 15). The Python repository tests on Python 3.9 to 3.13. We did not see CI results (7 of 10). 30 + 20 + 8 + 12 + 7 = 77.\n\n### Schema \u0026 documentation: fastCRW +8\n\n- fastCRW (91): OpenAPI 3.1 for 11 paths at version 0.41.0 in docs/openapi.json, checked against the server in CI, and typed JSON Schema inputs on every MCP tool (25). llms.txt (86 lines), llms-full.txt and Markdown sources for every docs page (10). Tool descriptions state purpose and how to chain them (crawl returns a job id to poll, map returns URLs only, extract needs an LLM), and the docs have a choose-an-endpoint page, but few descriptions say when not to use a tool (14 of 20). Inputs are typed with required fields and a renderer enum, though crawl and extract take a free-form JSON Schema object (12 of 15). Examples on every page and an errors page with each HTTP status, errorCode and next step (15). Semver releases with a dated CHANGELOG that flags breaking changes (15). 25 + 10 + 14 + 12 + 15 + 15 = 91.\n- Diffbot (83): The Extract APIs have a public OpenAPI 3.1.0 file, version 1.1.0, with 13 operations, and the reference pages are generated from it. We read that file and one rendered page. The descriptions for Crawl, Knowledge Graph and Web Search were not read (25). llms.txt exists but is a short index with no Markdown twins of the docs pages found (5 of 10). The Analyze API page has sections on when to use it and when not to (17 of 20). Parameters are typed, `url` is the one required field and the file has 15 enums, but `fields` is a comma-separated string and the file defines no shared response schemas (10 of 15). The file carries 24 examples. It documents only 200 and 500, with separate docs pages for 401, 404, 429 and 457 (11 of 15). The path is versioned `/v3` and a dated changelog with RSS runs from 2015 to 15 June 2026 (15). All per the 9 October check, not re-read on 4 November (terms bar automated access to the site). 25 + 5 + 17 + 10 + 11 + 15 = 83.\n\n### Transparency \u0026 trust: Diffbot +8\n\n- fastCRW (60): The engine and MCP server are AGPL-3.0 and the SDKs MIT, while the hosted control plane is closed under terms that name no legal entity (25 of 30). The terms say crawled data is handled transiently with zero data retention for approved accounts, but the privacy policy is undated, names no controller, gives no retention periods and mentions no DPA (10 of 30). The CHANGELOG flags breaking changes, deprecated v2 extract routes return a warning on every response, and the October 2026 402 change is dated in the docs, with no standing notice policy (10 of 20). Local data-source telemetry is disclosed with an opt-out. The privacy policy names Stripe, Google Analytics, Google Ads and an X pixel, but infrastructure and search providers are unnamed and no data location is given (8 of 20). 25 + 10 + 10 + 8 = 53.\n- Diffbot (68): A closed service under undated Terms of Use that cover the Site and the Service, cap liability at $10 and grant Diffbot a perpetual licence to process submitted queries. The clients and skills are MIT (13 of 30). The privacy policy of 29 August 2025 says query history and API calls are stored against the API key and kept no longer than necessary, with no retention period given. The GDPR and data rights page was not read (14 of 30). There is no deprecation policy. The changelog carries dated notices, such as a May 2024 entry announcing a field removal for 30 August 2024, and the terms give 14 days' notice of fee changes (8 of 20). The privacy policy names its service providers one by one, among them AWS, Stripe, Intercom, Salesforce, Slack and Amplitude, and says all are in the United States (14 of 20). All per the 9 October check, not re-read on 4 November. 13 + 14 + 8 + 14 = 49.\n\n### Reliability: Diffbot +5\n\n- fastCRW (50): Graded on the hosted lines, for the hosted API and MCP server an agent would call; the self-hosted engine would read higher on the local lines. fastcrw.com/status is a first-party page with a live health check and a 99.9% monthly uptime objective, but no components with history (10 of 20). It lists no incidents and keeps no history we could read (5). Concurrency per plan is published, 3 on Free to 150 on Scale, while the request-rate guard above it is deliberately unpublished (12 of 15). The rate-limits page documents Retry-After and X-RateLimit headers, gives exponential backoff code, and separates 429 (retry) from 402 (stop). No idempotency keys for crawl, extract or monitor starts (13 of 15). Pricing mentions an SLA only as 'Premier + SLA' on Enterprise, with no terms, and the terms promise no uptime (0). The hosted service is sold on paid plans with no beta label, though the engine is at 0.41.0 (10). 10 + 5 + 12 + 13 + 0 + 10 = 50.\n- Diffbot (55): Graded as a hosted API. status.diffbot.com is a Pingdom public report with 13 checks, among them Analyze API, Article API, Crawlbot and Websearch, with daily uptime for seven days and no incident notes (15 of 20). No 90-day record can be read. In the week to 9 October 2026 Analyze API shows 99.69 per cent, with 2, 11 and 17 minutes down on 4, 5 and 6 October, and Article API 5, 9 and 2 minutes on 5 to 7 October (5 of 30, the line for no readable history). Rate limits are published per plan, 5 calls a minute on Free, 5 a second on Startup and 25 a second on Plus (15). The 429 page says to pause one second and back off. No `Retry-After` header is documented, though the official client reads one (10 of 15). No SLA was found (0). The Extract APIs are at `/v3` with no beta label (10). All per the 9 October check, not re-read on 4 November: Diffbot's terms bar software that scrapes or crawls the Site or Service, so its site and status report weren't fetched. 15 + 5 + 15 + 10 + 0 + 10 = 55.\n\n## Score by category\n\n| Category | Weight | fastCRW | Diffbot | Edge |\n| --- | --- | --- | --- | --- |\n| Reliability | 16% (20 this run) | 50 | 55 | Diffbot +5 |\n| Performance | 10%, pending | pending | pending | not scored in this run |\n| Schema \u0026 documentation | 13% (16.2 this run) | 91 | 83 | fastCRW +8 |\n| Agent ergonomics | 13% (16.2 this run) | 91 | 63 | fastCRW +28 |\n| Security \u0026 auth | 14% (17.5 this run) | 40 | 21 | fastCRW +19 |\n| Payments \u0026 pricing | 10% (12.5 this run) | 35 | 40 | Diffbot +5 |\n| Task success | 10%, pending | pending | pending | not scored in this run |\n| Maintenance \u0026 community | 7% (8.8 this run) | 93 | 77 | fastCRW +16 |\n| Transparency \u0026 trust | 7% (8.8 this run) | 60 | 68 | Diffbot +8 |\n| Negative events | ≤15 | 0 | 0 | |\n| **Total** | | **64.3 · B** | **56.1 · C** | |\n\n## Facts side by side\n\n| Fact | fastCRW | Diffbot |\n| --- | --- | --- |\n| Kind | HTTP API | HTTP API |\n| Vendor | Nanofy OÜ | Diffbot Technologies Corp. |\n| Hosted endpoint | `https://fastcrw.com/mcp` | `https://api.diffbot.com/v3` |\n| Transports | stdio, Streamable HTTP | HTTP |\n| Auth | OAuth or key | API key |\n| Pricing | Freemium | Freemium |\n| x402 | no | no |\n| Licence | AGPL-3.0 | Proprietary service under Diffbot's Terms of Use. The Python client, the TypeScript client and the agent skills are MIT |\n| Tools exposed | 11 | none |\n| Read-only variant documented | no | no |\n| llms.txt | yes | yes |\n| MCP registry | `io.github.fastcrw/crw` | not listed |\n| Last release | 2026-11-04 | 2026-10-29 |\n| Terms last updated | 2026-04-04 | no date given |\n| Privacy policy last updated | no date given | 2025-08-29 |\n| Customer content may train models | not found in the text | not found in the text |\n| Terms restrict automated access | yes | yes |\n| Terms restrict benchmarking | yes | not found in the text |\n| Terms or service can change without notice | not found in the text | not found in the text |\n| Arbitration or class-action waiver | not found in the text | yes |\n| Popularity | 1.2k stars, 881 npm/wk | 9 npm/wk |\n\n## How the scores have moved\n\n### fastCRW\n\nNo score change since it was listed.\n\n### Diffbot\n\n- +1.8 on 4 November 2026: product\n\n## Verdicts\n\n**fastCRW.** Agents get 11 compact, annotated MCP tools with bounded output, a public OpenAPI spec, refunds on failed pages and a free self-hosted engine. The hosted service has no status history, publishes no SLA terms, names its operator only on the support page, and documents an MCP address with the API key in the URL path.\n\n**Diffbot.** Extract returns typed JSON for articles, products, discussions and other page types with no selectors to write, under a public OpenAPI 3.1 contract and a free plan that needs no card. The API token travels in the URL query string, tokens carry no scopes, and no security page, certification or security.txt was found.\n\n## Before you call either\n\n### fastCRW\n\n1. Connect to https://fastcrw.com/mcp with OAuth rather than the /mcp/\u003cAPI_KEY\u003e address, so the key stays out of URLs and logs\n2. Branch on `errorCode`. Stop on 402 insufficient_credits or llm_not_available, and on 429 wait for Retry-After before retrying\n3. Check `success` as well as the HTTP status, since an anti-bot wall answers 200 with `success: false` and `errorCode: anti_bot`\n4. Pass `startIndex` from a truncated result to read the rest of a long page instead of setting `maxLength: 0`\n5. Use native /v1 routes for new work; /firecrawl/v2 is for Firecrawl SDK migrations and keeps the older 429 for empty balances\n\n### Diffbot\n\n1. Call `GET https://api.diffbot.com/v3/analyze?url=\u003cencoded URL\u003e\u0026token=\u003ctoken\u003e` when the page type is unknown. Use `/v3/article` or `/v3/product` to force one schema\n2. Keep the token out of logs and shared URLs. It sits in the query string, so any proxy or log that records the request URL records the token\n3. Stay under 5 calls a minute on Free, 5 a second on Startup and 25 a second on Plus. On 429 wait at least one second and back off\n4. Check a 200 response for `errorCode`. The official client raises `ExtractionError` when a 200 reports a failed extraction\n5. Set `useProxy=default` only when a site blocks the fetch. A proxied page costs 2 credits against 1\n6. Install `diffbot` from PyPI, not `diffbot-python`, which stopped at 0.3.0\n\n## Questions\n\n### Which is better for AI agents, fastCRW or Diffbot?\n\nfastCRW scores 64.3 (B) on agent readiness against Diffbot's 56.1 (C), and leads in 4 of 7 scored categories. Diffbot leads on reliability, payments \u0026 pricing and transparency \u0026 trust.\n\n### Do fastCRW and Diffbot need an API key?\n\nfastCRW takes an API key or an OAuth sign-in. Diffbot needs an API key.\n\n### Can an agent call fastCRW and Diffbot without installing anything?\n\nYes. fastCRW has a hosted endpoint at https://fastcrw.com/mcp and Diffbot at https://api.diffbot.com/v3.\n\n### Are fastCRW and Diffbot open source?\n\nfastCRW is open source (AGPL-3.0). No open-source release is listed for Diffbot.\n\n\n## For agents\n\n- This comparison as JSON: https://www.anchorterminal.com/compare/crw-vs-diffbot.json, and with the fewest tokens: https://www.anchorterminal.com/compare/crw-vs-diffbot.min.md\n- Over MCP at https://www.anchorterminal.com/mcp (no key): `compare_tools {\"a\": \"crw\", \"b\": \"diffbot\"}`. From a terminal: `anchor compare crw diffbot`\n- Each listing in full: https://www.anchorterminal.com/api/v1/tools/crw.json and https://www.anchorterminal.com/api/v1/tools/diffbot.json\n\n## Other comparisons with fastCRW or Diffbot\n\n- [Bright Data vs fastCRW](https://www.anchorterminal.com/compare/bright-data-vs-crw.md)\n- [Bright Data vs Diffbot](https://www.anchorterminal.com/compare/bright-data-vs-diffbot.md)\n- [BrowserAct vs fastCRW](https://www.anchorterminal.com/compare/browseract-vs-crw.md)\n- [BrowserAct vs Diffbot](https://www.anchorterminal.com/compare/browseract-vs-diffbot.md)\n- [Context.dev vs fastCRW](https://www.anchorterminal.com/compare/context-dev-vs-crw.md)\n- [Context.dev vs Diffbot](https://www.anchorterminal.com/compare/context-dev-vs-diffbot.md)\n- [Crawl4AI vs fastCRW](https://www.anchorterminal.com/compare/crawl4ai-vs-crw.md)\n- [Crawl4AI vs Diffbot](https://www.anchorterminal.com/compare/crawl4ai-vs-diffbot.md)\n- [fastCRW vs Decodo Web Data API](https://www.anchorterminal.com/compare/crw-vs-decodo.md)\n- [fastCRW vs jevcrawl](https://www.anchorterminal.com/compare/crw-vs-jevcrawl.md)\n- [fastCRW vs Jina Reader](https://www.anchorterminal.com/compare/crw-vs-jina-reader.md)\n- [fastCRW vs Microlink](https://www.anchorterminal.com/compare/crw-vs-microlink.md)\n- [fastCRW vs Olostep](https://www.anchorterminal.com/compare/crw-vs-olostep.md)\n- [fastCRW vs Outcrawl](https://www.anchorterminal.com/compare/crw-vs-outcrawl.md)\n- [fastCRW vs Oxylabs Web Scraper API](https://www.anchorterminal.com/compare/crw-vs-oxylabs-web-scraper.md)\n- [fastCRW vs Scrape.do](https://www.anchorterminal.com/compare/crw-vs-scrape-do.md)\n- [fastCRW vs ScrapeGraphAI](https://www.anchorterminal.com/compare/crw-vs-scrapegraphai.md)\n- [fastCRW vs Scrapeless](https://www.anchorterminal.com/compare/crw-vs-scrapeless.md)\n- [fastCRW vs ScrapeOps Proxy API Aggregator](https://www.anchorterminal.com/compare/crw-vs-scrapeops.md)\n- [fastCRW vs ScraperAPI](https://www.anchorterminal.com/compare/crw-vs-scraperapi.md)\n- [fastCRW vs Scrapewise](https://www.anchorterminal.com/compare/crw-vs-scrapewise.md)\n- [fastCRW vs Scrapfly](https://www.anchorterminal.com/compare/crw-vs-scrapfly.md)\n- [fastCRW vs ScrapingAnt](https://www.anchorterminal.com/compare/crw-vs-scrapingant.md)\n- [fastCRW vs ScrapingBee](https://www.anchorterminal.com/compare/crw-vs-scrapingbee.md)\n- [fastCRW vs Scrapingdog](https://www.anchorterminal.com/compare/crw-vs-scrapingdog.md)\n- [fastCRW vs Spider](https://www.anchorterminal.com/compare/crw-vs-spider-cloud.md)\n- [fastCRW vs WebScraping.AI](https://www.anchorterminal.com/compare/crw-vs-webscraping-ai.md)\n- [fastCRW vs ZenRows](https://www.anchorterminal.com/compare/crw-vs-zenrows.md)\n- [fastCRW vs Zyte API](https://www.anchorterminal.com/compare/crw-vs-zyte.md)\n- [Decodo Web Data API vs Diffbot](https://www.anchorterminal.com/compare/decodo-vs-diffbot.md)\n- [Diffbot vs Jina Reader](https://www.anchorterminal.com/compare/diffbot-vs-jina-reader.md)\n- [Diffbot vs Scrape.do](https://www.anchorterminal.com/compare/diffbot-vs-scrape-do.md)\n- [Diffbot vs Scrapeless](https://www.anchorterminal.com/compare/diffbot-vs-scrapeless.md)\n- [fastCRW vs Firecrawl MCP](https://www.anchorterminal.com/compare/crw-vs-firecrawl-mcp.md)\n- [fastCRW vs Qovia](https://www.anchorterminal.com/compare/crw-vs-qovia.md)\n- [fastCRW vs ScrapeAtlas](https://www.anchorterminal.com/compare/crw-vs-scrapeatlas.md)\n- [fastCRW vs SoldFetch](https://www.anchorterminal.com/compare/crw-vs-soldfetch.md)\n- [Diffbot vs Firecrawl MCP](https://www.anchorterminal.com/compare/diffbot-vs-firecrawl-mcp.md)\n- [Diffbot vs jevcrawl](https://www.anchorterminal.com/compare/diffbot-vs-jevcrawl.md)\n- [Diffbot vs Microlink](https://www.anchorterminal.com/compare/diffbot-vs-microlink.md)\n- [Diffbot vs Olostep](https://www.anchorterminal.com/compare/diffbot-vs-olostep.md)\n- [Diffbot vs Outcrawl](https://www.anchorterminal.com/compare/diffbot-vs-outcrawl.md)\n- [Diffbot vs Oxylabs Web Scraper API](https://www.anchorterminal.com/compare/diffbot-vs-oxylabs-web-scraper.md)\n- [Diffbot vs Qovia](https://www.anchorterminal.com/compare/diffbot-vs-qovia.md)\n- [Diffbot vs ScrapeAtlas](https://www.anchorterminal.com/compare/diffbot-vs-scrapeatlas.md)\n- [Diffbot vs ScrapeGraphAI](https://www.anchorterminal.com/compare/diffbot-vs-scrapegraphai.md)\n- [Diffbot vs ScrapeOps Proxy API Aggregator](https://www.anchorterminal.com/compare/diffbot-vs-scrapeops.md)\n- [Diffbot vs ScraperAPI](https://www.anchorterminal.com/compare/diffbot-vs-scraperapi.md)\n- [Diffbot vs Scrapewise](https://www.anchorterminal.com/compare/diffbot-vs-scrapewise.md)\n- [Diffbot vs Scrapfly](https://www.anchorterminal.com/compare/diffbot-vs-scrapfly.md)\n- [Diffbot vs ScrapingAnt](https://www.anchorterminal.com/compare/diffbot-vs-scrapingant.md)\n- [Diffbot vs ScrapingBee](https://www.anchorterminal.com/compare/diffbot-vs-scrapingbee.md)\n- [Diffbot vs Scrapingdog](https://www.anchorterminal.com/compare/diffbot-vs-scrapingdog.md)\n- [Diffbot vs SoldFetch](https://www.anchorterminal.com/compare/diffbot-vs-soldfetch.md)\n- [Diffbot vs Spider](https://www.anchorterminal.com/compare/diffbot-vs-spider-cloud.md)\n- [Diffbot vs WebScraping.AI](https://www.anchorterminal.com/compare/diffbot-vs-webscraping-ai.md)\n- [Diffbot vs ZenRows](https://www.anchorterminal.com/compare/diffbot-vs-zenrows.md)\n- [Diffbot vs Zyte API](https://www.anchorterminal.com/compare/diffbot-vs-zyte.md)\n- [Apify MCP Server vs fastCRW](https://www.anchorterminal.com/compare/apify-mcp-vs-crw.md)\n- [Apify MCP Server vs Diffbot](https://www.anchorterminal.com/compare/apify-mcp-vs-diffbot.md)\n", "meta": { "attribution": "Anchor Terminal (https://www.anchorterminal.com)", "attributionRequired": true, "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)", "docs": "https://www.anchorterminal.com/docs/", "generatedAt": "2026-11-07", "license": "CC-BY-4.0", "method": "https://www.anchorterminal.com/benchmark/", "methodology": "0.7", "openapi": "https://www.anchorterminal.com/openapi.json", "preview": false, "run": "2026-10-01", "runLabel": "October 2026 research run", "terms": "https://www.anchorterminal.com/terms/#data" }, "page": { "breadcrumbs": [ { "name": "Home", "url": "https://www.anchorterminal.com/" }, { "name": "Compare", "url": "https://www.anchorterminal.com/compare/" }, { "name": "fastCRW vs Diffbot", "url": "" } ], "description": "fastCRW scores 64.3 (B) to Diffbot's 56.1 (C) for scraping. Prices, MCP, x402, uptime and agent notes side by side.", "facts": [ "fastCRW B 64.3", "Diffbot C 56.1", "scores" ], "h1": "fastCRW vs Diffbot", "image": "https://www.anchorterminal.com/assets/og/default.png", "path": "/compare/crw-vs-diffbot", "published": "2026-10-01", "section": "tools", "title": "fastCRW vs Diffbot for AI agents in 2026: scores and prices", "toc": null, "updated": "2026-11-07", "url": "https://www.anchorterminal.com/compare/crw-vs-diffbot" }, "tokens": { "markdown": 6250, "slim": 3580 }, "version": 1 }