Head to head · Scraping · October 2026 research run

ScraperAPI vs XCrawl

ScraperAPI scores 52.2 (D) on agent readiness against XCrawl's 30.4 (F), and leads in every scored category. Both do scraping.

Best web scraping and crawling APIs for AI agents · All 601 scrapers comparisons

Which one, for what

ScraperAPI D

Best for Agents that want a single GET to fetch pages through proxies, with parsed JSON for Amazon, Google, Walmart, eBay and Redfin and a batch API of up to 50,000 URLs.

Ahead on

  • Reliability, 42 against 15
  • Schema & documentation, 67 against 35
  • Agent ergonomics, 74 against 59
  • Security & auth, 29 against 18
  • Payments & pricing, 60 against 30
  • Maintenance & community, 41 against 17
  • Transparency & trust, 54 against 43

Watch for

No status page is linked from the home, pricing, support or docs pages, although the pricing page lists a 99.9% uptime guarantee

XCrawl F

Best for A general scrape, crawl, map and SERP API for agents, with 35 site-specific data endpoints.

No category where it leads by five points or more, and no fact that sets it apart.

Watch for

Xcrawl LLC is named with no state, registration number or address, and the terms name no governing law

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Schema & documentation ScraperAPI +32

ScraperAPI 67

No OpenAPI file found. The MCP server's source defines typed pydantic inputs for all 28 tools (12 of 25). The docs publish llms.txt, llms-full.txt and a Markdown twin of every page (10). MCP tool descriptions state when to use a tool and when not to, and the API reference describes each parameter with its credit cost (15 of 20). MCP inputs use enums for device type and output format. The HTTP API's parameters are query strings described in prose tables (10 of 15). Examples in cURL, Python, Node.js, PHP, Ruby and Java, a status code table that says which codes are billed, and an error table for the AI parser (12 of 15). Release notes are grouped by month without day dates, the HTTP API has no version, and the MCP server uses semver tags (8 of 15). 12 + 10 + 15 + 10 + 12 + 8 = 67.

XCrawl 35

No OpenAPI or other machine-readable spec found, and the MCP package is closed source, so its tool schemas couldn't be read (0 of 25). llms.txt returns 404. The docs have "Copy page" buttons, and the xcrawl-skills repository has Markdown skill files for agents (5 of 10). Each API reference page states what the endpoint does, with no guidance on when not to use it (8 of 20). Parameter tables give types, required flags, defaults and enums for formats, device and wait_until. cookies and headers are free maps (11 of 15). Request and response examples on every reference page, but no error responses or codes documented outside the MCP troubleshooting table (6 of 15). Versioned /v1 paths and no changelog (5 of 15). Total 35.

Payments & pricing ScraperAPI +30

ScraperAPI 60

No x402, MPP or L402 found in the docs index or pricing page (0). Plan prices and credit costs per request type are public without a login (20). Free plan of 1,000 credits a month and a 7-day trial of 5,000 credits, no card (20). A person signs up in a browser to get a key (0). 0 + 20 + 20 + 0 = 40. Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 40 is 60.

XCrawl 30

No x402, MPP or L402 (0). The pricing page's plan table and credit table did not appear in the served page, and the docs give credit costs only in examples (1 for a plain scrape, 2 for a search), so pricing is scored absent (0 of 30, unchecked). 1,000 free credits on signup with no card (30). A person signs up in a browser at dash.xcrawl.com to get a key (0). Total 30.

Reliability ScraperAPI +27

ScraperAPI 42

No status page is linked from the home, pricing, support or docs pages we read on 9 October or 4 November (0). No readable incident history (5). Concurrency limits are published per plan, 5 threads on Free, 20 on Hobby, 50, 100, 200, 300 and 500 above it, and the local MCP server limits itself to 10 calls in 60 seconds (15). 429 is documented as a concurrency rejection with no charge, the API retries a target for up to 70 seconds before returning 500, failed requests aren't charged, and the docs tell clients to retry failed requests. That is safe-retry guidance with no idempotency keys on the async job writes and no Retry-After header (12 of 15). The pricing page lists a 99.9% uptime guarantee on every plan, but no SLA document or credit terms were found and the Terms of Use are closed to us (0 of 10). The API, async service and crawler are generally available (10). 0 + 5 + 15 + 12 + 0 + 10 = 42.

XCrawl 15

No status page found on xcrawl.com or docs.xcrawl.com (0 of 20) and no readable incident history (5 of 30). No rate limits with numbers. The pricing page says only that Scale plans get higher rate limits (0 of 15). No 429 or overload guidance in the docs. The Python SDK retries 500, 502, 503 and 504 with backoff, but not 429 (0 of 15). The terms mention a Service Level Agreement "if applicable", and none is published (0 of 10). The API is sold publicly with no beta label (10). Total 15.

Maintenance & community ScraperAPI +24

ScraperAPI 41

The newest dated entry is the October 2026 release note on automatic Akamai unblocking, dated by month only. Counted from its month, October lies 4 to 34 days back, so it is inside 90 days but not shown within 30, as crustdata's October entries. The skills plugin's 1.1.0 of 7 August is an untagged version bump, and the MCP server's v1.1.1 is from 17 July 2026 (20). Since 6 August only the October note and the 7 August plugin bump are dated, fewer than three (0 of 20). The npm and PyPI pages of the SDKs and CLI couldn't be read on 4 November. Monthly release notes and support through the dashboard and email. We did not read the repository's issues (8 of 15). SDKs are documented for five languages, but we could not check their release dates, and the Node.js SDK repository's last commit is from 12 January 2024 (8 of 15). The MCP repository has unit tests and a lock file but no public CI workflow (5 of 10). 20 + 0 + 8 + 8 + 5 = 41.

XCrawl 17

The newest dated releases are the Python SDK 1.1.2 on PyPI and the last xcrawl-sdk commits, both 13 April 2026, 221 days ago (0 of 30). Nothing dated in the 90 days from 22 August 2026 (0 of 20). We couldn't read GitHub issues. There is no public changelog, and support is by email, Telegram, WhatsApp and Microsoft Teams, with a 24-hour reply promised in the footer (5 of 15). Official Python and Node.js SDKs, unchanged for seven months (10 of 15). The SDK repository has 3 commits, no tests and no CI. Dependencies are current (2 of 10). Total 17.

Agent ergonomics ScraperAPI +15

ScraperAPI 74

28 MCP tools with no toolsets or dynamic loading (15 of 25). output_format returns Markdown or text, autoparse returns JSON on supported domains, structured endpoints page their results, and max_cost, crawl_budget and max_depth cap a job (15 of 20). Errors are HTTP status codes with plain-text messages. No machine-readable error codes (13 of 20). All 28 MCP tools carry readOnlyHint, destructiveHint, idempotentHint and openWorldHint, failed requests are not charged, and async jobs return a status URL to poll (16 of 20). Only api_key and url are required, rendering is off by default, and SDKs are documented for Python, Node.js, PHP, Ruby and Java (15). 15 + 15 + 13 + 16 + 15 = 74.

XCrawl 59

7 MCP tools per the Claude Code guide, compact by name and description, though the definitions themselves couldn't be read (22 of 25). Output formats, only_main_content, a search limit of 1 to 100, and crawl limit, depth and include or exclude regex keep responses sized (16 of 20). The SDKs raise errors with a code, status and request_id, but the docs list no error codes or recovery steps (5 of 20). Async jobs return IDs for polling and webhooks, with no idempotency keys or MCP annotations found (3 of 20). Only url or query is required, and Python and Node.js SDKs exist, but skip_tls_verification defaults to true (13 of 15). Total 59.

Security & auth ScraperAPI +11

ScraperAPI 29

One API key per account, regenerated from the dashboard at most once every 24 hours. No scopes (20). The documented default puts the key in the URL query string, so less 10, although an x-sapi-api_key header and a Bearer header on the remote MCP server exist (10 of 30). No read-only or restricted keys. max_cost, a monthly pay-as-you-go spending cap and destructive hints on the two MCP delete tools limit damage (6 of 20). The API returns untrusted web content and no prompt-injection guidance was found in the docs, the MCP README or the skills (2 of 15). The dashboard has analytics with error logs, each response carries sa-credit-cost, and the privacy policy says activity logs are deleted after 7 days (9 of 15). /.well-known/security.txt returned 404 on 9 October and 4 November, and no disclosure policy, bug bounty or certification was found. The home page claims CCPA and GDPR compliance (2 of 20). 10 + 6 + 2 + 9 + 2 = 29.

XCrawl 18

A plain bearer API key from the dashboard with no scopes, and the MCP guide says to reissue an invalid key there (20). The recommended hosted MCP setup puts the key in the URL path, so less 10, read the way the checklist reads a key in a query string (10 of 30). The API only reads third-party pages and has no destructive account actions, but there is no read-only or least-privilege key (5 of 20). No prompt-injection guidance in the docs or the skills (0 of 15). Each response reports credits used, and xcrawl_key_status gives the balance, with no request log found (3 of 15). No security.txt, disclosure policy or bug bounty. The ISO 27001 badge on the home page links to no certificate or security page (0 of 20). Total 18.

Score by category

CategoryWeightScraperAPIXCrawlEdge
Reliability16%4215ScraperAPI +27
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%6735ScraperAPI +32
Agent ergonomics13%7459ScraperAPI +15
Security & auth14%2918ScraperAPI +11
Payments & pricing10%6030ScraperAPI +30
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%4117ScraperAPI +24
Transparency & trust7%5443ScraperAPI +11
Negative events≤1500
TotalD 52.2/100F 30.4/100

Facts side by side

FactScraperAPIXCrawl
KindHTTP APIHTTP API
VendorScraperAPI (saas.group)XCrawl
Hosted endpointhttps://api.scraperapi.comhttps://mcp.xcrawl.com/{YOUR_API_KEY}/mcp
TransportsHTTP, Streamable HTTP, stdioStreamable HTTP, stdio
AuthAPI keyAPI key
PricingFreemiumFreemium
x402nono
Tools exposed287
Read-only variant documentedyesno
llms.txtyesno
Last release2026-08-072026-04-13
Terms last updatedcouldn't be readno date given
Privacy policy last updatedno date givenno date given
Customer content may train modelscouldn't be readnot found in the text
Terms restrict automated accesscouldn't be readnot found in the text
Terms restrict benchmarkingcouldn't be readnot found in the text
Terms or service can change without noticecouldn't be readyes
Arbitration or class-action waivercouldn't be readnot found in the text
Popularity5 starsnone

How the scores have moved

Verdicts

ScraperAPI

One GET request with a key and a URL returns a page, and the MCP server's 28 tools carry read-only and destructive hints. No status page, OpenAPI file, security.txt or certification was found, and the documented default puts the API key in the URL query string.

XCrawl

A scrape, search, map and crawl API with 7 MCP tools, MIT Python and Node.js SDKs and 1,000 free credits without a card. Xcrawl LLC gives no address or registration. There is no OpenAPI spec, status page, rate limit, error reference or changelog, plan prices did not render for us, and the hosted MCP key travels in the URL.

Before you call either

ScraperAPI

  1. Set the client timeout to 70 seconds. The API retries for that long before it returns 500, and a request cancelled earlier is still charged
  2. Send the key as the x-sapi-api_key header to keep it out of URLs and logs
  3. Pass max_cost on each request, or call /account/urlcost first, because domain and anti-bot surcharges vary from 1 to 75 credits
  4. Use output_format=markdown or text to cut response size, and autoparse=true only on supported domains
  5. On 429 reduce concurrency to the plan limit (5 on Free, 20 on Hobby). The request is rejected, not queued

XCrawl

  1. Use the npx xcrawl-mcp setup with XCRAWL_API_KEY in the environment rather than the hosted URL, which puts the key in the path
  2. Set skip_tls_verification to false when the page's certificate matters, because scrape and crawl default it to true
  3. Set crawler.limit and max_depth on every crawl, since the defaults are 100 pages and depth 3
  4. Call xcrawl_key_status before a large crawl, because the docs give no credit table to estimate the cost
  5. Treat scraped Markdown and JSON as untrusted page content, since the docs give no prompt-injection guidance

Questions

Which is better for AI agents, ScraperAPI or XCrawl?

ScraperAPI scores 52.2 (D) on agent readiness against XCrawl's 30.4 (F), and leads in every scored category.

Do ScraperAPI and XCrawl need an API key?

Both need an API key.

Can an agent call ScraperAPI and XCrawl without installing anything?

Yes. ScraperAPI has a hosted endpoint at https://api.scraperapi.com and XCrawl at https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp.

Other comparisons with ScraperAPI or XCrawl

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.