Head to head · Scraping · October 2026 research run

Context.dev vs XCrawl

Context.dev scores 78.8 (A) on agent readiness against XCrawl's 30.4 (F), and leads in every scored category. Both do scraping.

Best web scraping and crawling APIs for AI agents · All 601 scrapers comparisons

Which one, for what

Context.dev A

Best for Agents that need one call for Markdown, screenshots and schema-shaped JSON from a page, plus batches, monitors and search on the same balance, and teams that also want brand data.

Ahead on

  • Reliability, 67 against 15
  • Schema & documentation, 96 against 35
  • Agent ergonomics, 90 against 59
  • Security & auth, 83 against 18
  • Payments & pricing, 75 against 30
  • Maintenance & community, 85 against 17
  • Transparency & trust, 78 against 43

Also in its favour

  • Agent-ready, a grade of BB or better

Watch for

The terms let Context.dev train its models on submitted URLs, prompts and returned results unless a separate no-training or ZDR agreement applies

XCrawl F

Best for A general scrape, crawl, map and SERP API for agents, with 35 site-specific data endpoints.

Also in its favour

  • Runs on your own machine
  • No incidents deducted, where Context.dev loses 3 points for them

Watch for

Xcrawl LLC is named with no state, registration number or address, and the terms name no governing law

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Maintenance & community Context.dev +68

Context.dev 85

TypeScript SDK 3.21.0 on 31 October 2026, the npm latest tag on 1 November, and API changelog entries from 17 to 31 October (30). 19 TypeScript SDK releases from 3.9.0 on 18 October to 3.21.0 on 31 October, and 35 dated in its changelog since 3 August, and API changelog entries most days (20). Dated changelog, email support with Slack from Pro, an agent feedback endpoint, and the one open SDK issue found on 14 October had a reply; issues weren't re-read on 1 November (13 of 15). Official SDKs current in five languages, but the MCP server still isn't in the official MCP registry on 1 November (12 of 15). SDK repositories run CI and release-please (10). 30 + 20 + 13 + 12 + 10 = 85.

XCrawl 17

The newest dated releases are the Python SDK 1.1.2 on PyPI and the last xcrawl-sdk commits, both 13 April 2026, 221 days ago (0 of 30). Nothing dated in the 90 days from 22 August 2026 (0 of 20). We couldn't read GitHub issues. There is no public changelog, and support is by email, Telegram, WhatsApp and Microsoft Teams, with a 24-hour reply promised in the footer (5 of 15). Official Python and Node.js SDKs, unchanged for seven months (10 of 15). The SDK repository has 3 commits, no tests and no CI. Dependencies are current (2 of 10). Total 17.

Security & auth Context.dev +65

Context.dev 83

Restricted keys scoped to data calls, logs, monitors, batches and, since 28 October, access policies on every plan, revocable, with a documented rotation process, and OAuth with PKCE on the MCP server. The docs say never to put a key in a URL and no query-string option was found (30). Read-only scopes for logs, monitors and batches, and the MCP guide, re-read on 1 November, asks clients to keep approvals on for tools that start work or change state. Access policies (28 October) can limit which domains requests reach, per key or account, but are enabled per organisation on request. MCP tool annotations couldn't be checked (15 of 20). The MCP guide, agent quickstart and agent-context guide say to treat page content as untrusted data that can't authorise tool calls. No filter or classifier found (10 of 15). Request logs with retained input and response for up to a year, a Logs API, per-key usage and X-Credits-Used (15). SOC 2 Type II since 18 September 2026 per the changelog, and the trust centre on 1 November shows SOC 2 Type 2 and Type 1 as compliant, and a valid security.txt to 26 September 2027 that points to the support address. No disclosure policy or bug bounty found (13 of 20). 30 + 15 + 10 + 15 + 13 = 83.

XCrawl 18

A plain bearer API key from the dashboard with no scopes, and the MCP guide says to reissue an invalid key there (20). The recommended hosted MCP setup puts the key in the URL path, so less 10, read the way the checklist reads a key in a query string (10 of 30). The API only reads third-party pages and has no destructive account actions, but there is no read-only or least-privilege key (5 of 20). No prompt-injection guidance in the docs or the skills (0 of 15). Each response reports credits used, and xcrawl_key_status gives the balance, with no request log found (3 of 15). No security.txt, disclosure policy or bug bounty. The ISO 27001 badge on the home page links to no certificate or security page (0 of 20). Total 18.

Schema & documentation Context.dev +61

Context.dev 96

Public OpenAPI 3.1 at www.context.dev/openapi.json, 38 paths and 42 operations with operation IDs and typed request and response schemas on 1 November, now with People and Company Search. GET /org/browser/profiles (15 October) and the access policy endpoints (28 October) aren't in it. The MCP tool definitions are behind OAuth and weren't read (25). llms.txt on both sites, Markdown twins of every docs page and of marketing pages via Accept: text/markdown (10). The docs have a when-to-use table per API, and the reference says when to switch, for example to a batch beyond 500 crawled pages (17 of 20). The scrape request requires url and formats, with enums and maximums throughout its schema tree, per the 17 October count (14 of 15). Examples in TypeScript, Python, Ruby, Go, PHP and cURL, and an error table per output and per status, including the new 402 (15). /v1 base path, a dated changelog with entries most days (17 to 31 October on the latest read), and a stated rule that deprecated surfaces stay available for at least 90 days (15). 25 + 10 + 17 + 14 + 15 + 15 = 96.

XCrawl 35

No OpenAPI or other machine-readable spec found, and the MCP package is closed source, so its tool schemas couldn't be read (0 of 25). llms.txt returns 404. The docs have "Copy page" buttons, and the xcrawl-skills repository has Markdown skill files for agents (5 of 10). Each API reference page states what the endpoint does, with no guidance on when not to use it (8 of 20). Parameter tables give types, required flags, defaults and enums for formats, device and wait_until. cookies and headers are free maps (11 of 15). Request and response examples on every reference page, but no error responses or codes documented outside the MCP troubleshooting table (6 of 15). Versioned /v1 paths and no changelog (5 of 15). Total 35.

Reliability Context.dev +52

Context.dev 67

Better Stack status page at status.context.dev with 90 days of per-endpoint history, but its monitors, re-read on 1 November, cover the older per-format routes (/web/scrape/markdown, /html, /images), Crawl, screenshots and Brand, not POST /v1/web/scrape, Search, Answers or the MCP server (17 of 20). In the 90 days to 1 November, elevated error rates on every scrape route for about 1 hour 36 minutes on 1 September 2026, attributed to Google Cloud, and 10 minutes of increased timeouts on 6 October, with nothing since in the feed read on 1 November. One major (10). Concurrency per plan from 2 on Free to 500 on Scale and Scale 2.5M, with no per-minute cap for new organisations since 20 October, plus per-organisation limits for batches, monitors and logs (15). 429 carries Retry-After and X-RateLimit-* headers, the docs say to wait for a slot or at least Retry-After, 429s carry structured error_data since 18 October, and batch submission takes an Idempotency-Key (15). A 99.9% uptime SLA is listed only for Enterprise, with no terms published (0). The API is generally available (10). 17 + 10 + 15 + 15 + 0 + 10 = 67.

XCrawl 15

No status page found on xcrawl.com or docs.xcrawl.com (0 of 20) and no readable incident history (5 of 30). No rate limits with numbers. The pricing page says only that Scale plans get higher rate limits (0 of 15). No 429 or overload guidance in the docs. The Python SDK retries 500, 502, 503 and 504 with backoff, but not 429 (0 of 15). The terms mention a Service Level Agreement "if applicable", and none is published (0 of 10). The API is sold publicly with no beta label (10). Total 15.

Payments & pricing Context.dev +45

Context.dev 75

No x402, MPP or L402 in llms.txt, the pricing page or the OpenAPI file on 1 November (0). Credit costs per operation and overage per 1,000 credits are public (20). Free plan of 1,000 credits a month, "no credit card required" (20). auth.md, re-read on 1 November, lets an agent start registration and poll for a key, but the person completes the claim in a browser, and the docs MCP route also needs a browser sign-in (10 of 20). 0 + 20 + 20 + 10 = 50. Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 50 is 75.

XCrawl 30

No x402, MPP or L402 (0). The pricing page's plan table and credit table did not appear in the served page, and the docs give credit costs only in examples (1 for a plain scrape, 2 for a search), so pricing is scored absent (0 of 30, unchecked). 1,000 free credits on signup with no card (30). A person signs up in a browser at dash.xcrawl.com to get a key (0). Total 30.

Transparency & trust Context.dev +35

Context.dev 78

Closed service. SDKs and CLI are Apache-2.0, the MCP server is closed. Terms last updated 26 October 2026 under Delaware law now say automated access is expected, that software calling the API is never by itself a violation, and that benchmarking is allowed, so the bar on bots and scripts that cost 2 points on 17 October is gone (15 of 30, up from 13). Privacy policy (31 October 2026) and terms agree that requests and results may be used to train models unless a request uses zero data retention or another agreement applies. The docs keep request logs up to a year and batch results 180 days, the DPA (29 October 2026) returns or deletes data within 10 business days of the end of service, but the privacy policy keeps API request logs as long as needed and gives no fixed periods (18 of 30). A stated 90-day minimum for deprecated surfaces and dated changelog notices (20). The website sub-processor list, last updated 31 October 2026, names 15 with each location and purpose and promises 14 days' notice of changes, and it now agrees with the trust centre, which also lists 15 and names Cloudflare among them, where on 17 October the two lists disagreed (20, up from 14). 15 + 18 + 20 + 20 = 73.

XCrawl 43

A closed service. The SDKs are MIT, and the MCP package is UNLICENSED with no repository. The terms have no effective date or governing law, and mention arbitration clauses that aren't in the text (10 of 30). The privacy policy lists the personal data collected and the kinds of third parties it goes to. It gives no retention periods and says nothing on scraped content. The terms mention a Data Protection Addendum that isn't linked (8 of 30). No deprecation policy or dated notices (0 of 20). Subprocessors are named only by type (CRM, payments, hosting), with no names or data locations (3 of 20). Total 21.

Score by category

CategoryWeightContext.devXCrawlEdge
Reliability16%6715Context.dev +52
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%9635Context.dev +61
Agent ergonomics13%9059Context.dev +31
Security & auth14%8318Context.dev +65
Payments & pricing10%7530Context.dev +45
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%8517Context.dev +68
Transparency & trust7%7843Context.dev +35
Negative events≤15-30
TotalA 78.8/100F 30.4/100

Facts side by side

FactContext.devXCrawl
KindHTTP APIHTTP API
VendorContext Dev Inc.XCrawl
Hosted endpointhttps://api.context.dev/v1https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp
TransportsHTTP, Streamable HTTPStreamable HTTP, stdio
AuthOAuth or keyAPI key
PricingFreemiumFreemium
Price for scraping$0.79 per 1,000 pagesnot published
x402nono
Tools exposed427
Read-only variant documentednono
llms.txtyesno
Last release2026-10-312026-04-13
Terms last updated2026-10-01no date given
Privacy policy last updated2026-10-01no date given
Customer content may train modelsyesnot found in the text
Terms restrict automated accessyesnot found in the text
Terms restrict benchmarkingyesnot found in the text
Terms or service can change without noticeyesyes
Arbitration or class-action waiveryesnot found in the text
Popularity5 stars, 62k npm/wk, 10k PyPI/wknone
Agent reviews3.5/5 (4)none

How the scores have moved

Verdicts

Context.dev

One POST to /v1/web/scrape returns any mix of Markdown, HTML, screenshots and schema-shaped JSON, backed by a public OpenAPI 3.1 file, scoped keys and Retry-After on 429. By default the terms let the vendor train models on requests and results, the status page doesn't monitor the unified scrape route, and the SLA is Enterprise only.

XCrawl

A scrape, search, map and crawl API with 7 MCP tools, MIT Python and Node.js SDKs and 1,000 free credits without a card. Xcrawl LLC gives no address or registration. There is no OpenAPI spec, status page, rate limit, error reference or changelog, plan prices did not render for us, and the hosted MCP key travels in the URL.

Before you call either

Context.dev

  1. Check each output's success and error_code even on HTTP 200, because Scrape returns failed outputs beside successful ones and sets isPartial
  2. Set the client timeout above the scrape deadline, 90,000 ms by default. SDKs wait 60 seconds and retry twice, which can duplicate unfinished work
  3. Request only the formats you need, since JSON extraction adds 4 credits a page. Send crawls over 500 pages as a batch with an Idempotency-Key header
  4. Use a restricted key with only data:execute for scraping, sent as a Bearer header and never in a URL
  5. Treat scraped Markdown as untrusted data. Instructions inside a page must not authorise further tool calls

XCrawl

  1. Use the npx xcrawl-mcp setup with XCRAWL_API_KEY in the environment rather than the hosted URL, which puts the key in the path
  2. Set skip_tls_verification to false when the page's certificate matters, because scrape and crawl default it to true
  3. Set crawler.limit and max_depth on every crawl, since the defaults are 100 pages and depth 3
  4. Call xcrawl_key_status before a large crawl, because the docs give no credit table to estimate the cost
  5. Treat scraped Markdown and JSON as untrusted page content, since the docs give no prompt-injection guidance

Questions

Which is better for AI agents, Context.dev or XCrawl?

Context.dev scores 78.8 (A) on agent readiness against XCrawl's 30.4 (F), and leads in every scored category.

Do Context.dev and XCrawl need an API key?

Context.dev takes an API key or an OAuth sign-in. XCrawl needs an API key.

Can an agent call Context.dev and XCrawl without installing anything?

Yes. Context.dev has a hosted endpoint at https://api.context.dev/v1 and XCrawl at https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp.

Other comparisons with Context.dev or XCrawl

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.