Best of · Web, data & documents
Best web scraping and crawling APIs for AI agents
The 10 highest-scoring of 18 web scraping and crawling APIs on the Anchor benchmark, with a pick for each need and where each one falls short. Scores come from public evidence, re-checked as vendors change.
- 18 ranked
- 4 agent-ready
- 2 accept x402
- 18 hosted endpoints
- Updated 9 October 2026
Top three
Picks by need
Worked out from the scores, prices and facts, so they change when the research does.
Schema & documentation
Scrapfly B
95/100 on schema & documentation, against 92 for the overall leader.
Lowest paid price per 1,000 requests
Zyte API B
$0.06 per 1,000 requests, the lowest of the 14 listings here with a paid price in this unit (free allowances aside).
Also Scrape.do, $0.116 per 1,000 requests.
Paying per call with no account (x402)
accepts x402, $1 a call.
Also Spider, accepts x402, $0.0005 a call.
The shortlist
| # | Tool | Grade | Best for | Price | Where |
|---|---|---|---|---|---|
| 1 | Apify MCP Server Apify |
BB 76.5 | Best when the job matches an existing Actor (Google Maps, social profiles, e-commerce, SERP) and the agent can wait seconds to minutes for a run. | $1 / call | hosted and local |
| 2 | Firecrawl MCP Firecrawl (Mendable) |
BB 75.3 | Agents that need the whole pipeline (find URLs, scrape to markdown, crawl, monitor) from one server, and for trying without an account. | $19 / mo | hosted and local |
| 3 | ZenRows ZenRows |
BB 74.8 | An agent hitting protected sites that wants the service to choose the setup and bill only success, and for agents that must onboard without a human. | $19 / mo | hosted and local |
| 4 | Spider Spider (BAGELMEN LLC) |
BB 74.1 | An agent that has to start without a human, pay per call, or crawl many small pages cheaply. | $0.0005 / call | hosted and local |
| 5 | Zyte API Zyte Group Limited |
B 69.7 | An agent or pipeline that fetches protected pages and wants the service to choose proxy and browser per site and to bill only success, and teams already on Scrapy. | $0.13 / 1k req | hosted |
| 6 | Scrapfly Scrapfly (Joam Intelligence, LLC) |
B 69.6 | An agent that needs scrape, screenshot, extraction, crawl and a hosted browser from one key, with errors it can branch on. | $30 / mo | hosted and local |
| 7 | ScrapeGraphAI ScrapeGraphAI |
B 68.3 | LLM pipelines that want prompt-to-JSON extraction and for agents that must pay with a wallet. | $20 / mo | hosted |
| 8 | Olostep Olostep Technologies, Inc. |
B 63 | Bulk scraping on a budget and for batch jobs of thousands of URLs. | $9 / mo | hosted and local |
| 9 | Oxylabs Web Scraper API oxylabs, UAB |
B 63 | Agents that need parsed Google, Amazon and other large-site results or bulk asynchronous jobs through one API. | $30 / mo | hosted and local |
| 10 | Bright Data Bright Data |
C 59.8 | Hard targets and site-specific structured data (Amazon, LinkedIn, Instagram, Google Maps) at volume. | $1.50 / 1k req | hosted |
8 more are ranked in the full table.
How to choose
- Cost per successful requestCheck whether failed and blocked requests are billed, because the list price per request can understate what each usable page costs an agent.
- Success rate on protected sitesAsk for success rates on sites behind anti-bot checks, since a service that only handles open pages will fail on the targets an agent most often needs.
- Output format and size capConfirm the output formats and the response size cap, because a page cut short at the cap can drop text an agent needs without any error.
- Terms for reusing scraped outputRead the terms on storing, reusing and redistributing scraped output, since some services limit how long the agent may keep what it collects.
Each one in detail
Apify MCP Server
BB 76.5/100Exposes thousands of Apify Store Actors (scrapers, crawlers, automations) as dynamically discovered MCP tools, hosted at mcp.apify.com or run locally; supports OAuth, API tokens and agentic payments (x402 prepaid tokens, Skyfire).
Verdict x402 on Apify's own domains, a prepaid token from agi.apify.com for any Actor and per-run payment on mcp.apify.com for Pay Per Event Actors. API and Actor runs timed out for about 12 hours on 21 and 22 July 2026.
Choose it for Best when the job matches an existing Actor (Google Maps, social profiles, e-commerce, SERP) and the agent can wait seconds to minutes for a run.
Strengths
- x402 on Apify's own domains, a prepaid token from agi.apify.com for any Actor and per-run payment on mcp.apify.com for Pay Per Event Actors
- 12 tools by default out of 35,
?tools=to pick categories, and readOnly, destructive and idempotent hints on every tool - Scoped API tokens with expiry and a 24-hour rotation overlap, plus OAuth on the hosted server
Weaknesses
- API and Actor runs timed out for about 12 hours on 21 and 22 July 2026
- No prompt-injection guidance for scraped content or third-party Actor READMEs
- Telemetry and Sentry on by default
Price $1 / callAuth OAuth or keyx402 yeshosted and local
Firecrawl MCP
BB 75.3/100Official Firecrawl server for scraping pages to markdown/JSON, mapping and crawling sites, web search, browser interaction, monitoring and an autonomous research agent.
Verdict Keyless hosted endpoint with scrape, search and parse, plus OAuth and a search-only endpoint. No x402 or other machine payment.
Choose it for Agents that need the whole pipeline (find URLs, scrape to markdown, crawl, monitor) from one server, and for trying without an account.
Strengths
- Keyless hosted endpoint with scrape, search and parse, plus OAuth and a search-only endpoint
- Four incidents since July 2026, the longest 56 minutes
- Rate limits published per plan and per endpoint
Weaknesses
- No x402 or other machine payment
- No prompt-injection guidance for scraped pages outside retained Alexandria results
- 403 and 404 responses cost a credit
Price $19 / moAuth OAuth or keyx402 nohosted and local
ZenRows
BB 74.8/100Scraping API (Fetch, formerly Universal Scraper API) with JavaScript rendering, residential proxies and anti-bot bypass, plus Extract for structured JSON, Batch for bulk URL jobs and hosted Browser Sessions.
Verdict No incidents from July to 1 October 2026 on a six-component Better Stack status page. The Fetch API only takes the key in the query string.
Choose it for An agent hitting protected sites that wants the service to choose the setup and bill only success, and for agents that must onboard without a human.
Strengths
- No incidents from July to 1 October 2026 on a six-component Better Stack status page
- 5,000 free credits a month with no card, and a stdio MCP that can provision a free account by itself
- Every MCP tool carries readOnlyHint and destructiveHint, and the scrape tool says when to use extract instead
Weaknesses
- The Fetch API only takes the key in the query string
- A protected request costs 25 credits, so real costs sit well above the headline rate
- 44 MCP tools load at once, 36 of them for the browser
Price $19 / moAuth OAuth or keyx402 partialhosted and local
Spider
BB 74.1/100Hosted crawl, scrape, search, links and screenshot API built on the open-source Rust spider crate, plus a cloud browser and proxy pool.
Verdict Keyless /scrape and working x402 v2 on every core route, so an agent needs no account. Operated by BAGELMEN LLC with no address, retention periods or DPA in the privacy policy.
Choose it for An agent that has to start without a human, pay per call, or crawl many small pages cheaply.
Strengths
- Keyless /scrape and working x402 v2 on every core route, so an agent needs no account
- Metered by bytes and CPU, so small pages cost a fraction of a cent
- Published rate limits with RateLimit and Retry-After headers
Weaknesses
- Operated by BAGELMEN LLC with no address, retention periods or DPA in the privacy policy
- No security.txt, disclosure policy, certification or prompt-injection guidance found
- Invalid parameter values fall back silently instead of returning 400
Price $0.0005 / callAuth OAuth or keyx402 yeshosted and local
Zyte API
B 69.7/100Zyte API is a hosted web scraping API from Zyte Group Limited of Cork, Ireland. One endpoint returns a page's HTTP body, browser-rendered HTML, a screenshot or extracted structured data, with ban avoidance handled by the service.
Verdict One POST endpoint with an OpenAPI 3 contract picks proxies, browser rendering and extraction per site, bills only successful responses, and sends problem+json errors with Retry-After. Prices vary by target site across five tiers, so cost per request isn't known before the site is assigned one, and no dated API changelog was found.
Choose it for An agent or pipeline that fetches protected pages and wants the service to choose proxy and browser per site and to bill only success, and teams already on Scrapy.
Strengths
- OpenAPI 3.0.3 contract for
/v1/extractprinted in the reference, with 51 enums and request examples, plus llms.txt and Markdown twins of the docs pages - Only successful responses are billed. Rate-limited, banned and failed requests are free, per the pricing page and the terms
- 429, 503 and 520 responses carry
Retry-After, and the docs publish a backoff table for each retry
Weaknesses
- Request price depends on the target site's tier, from $0.13 to $1.27 per 1,000 HTTP requests and $1.01 to $16.08 with a browser on pay-as-you-go
- No dated changelog or deprecation policy for the API was found in the 150-page docs sitemap, and the terms allow changes with or without notice
- The terms let Zyte use Service Data, data feeds and code from the services for product development and product training
Price $0.13 / 1k reqAuth OAuth or keyx402 nohosted
Scrapfly
B 69.6/100Scrape API with datacenter or residential proxies, JS rendering and an anti-bot Unblocker, plus screenshot, LLM extraction, crawler and hosted Cloud Browser APIs on one credit balance.
Verdict The error catalogue documents HTTP status, retry behaviour and billing for over 100 codes. The MCP server loads 61 tools at once.
Choose it for An agent that needs scrape, screenshot, extraction, crawl and a hosted browser from one key, with errors it can branch on.
Strengths
- Error catalogue of 100+ ERR:: codes, each with HTTP status, retryable and billed flags and a doc link
- Hosted MCP with OAuth open to all plans, and API keys restrictable to products since 15 April 2026
- MCP tools carry readOnlyHint, destructiveHint and idempotentHint, and mutating vault and alert tools need confirm=true
Weaknesses
- The open-source MCP server registers 61 tools at once, a heavy tools/list
- The canonical API auth form and one MCP option put the key in the URL query string
- Status page shows an overall state only, with no components or incident history, and status.scrapfly.io doesn't resolve
Price $30 / moAuth OAuth or keyx402 nohosted and local
ScrapeGraphAI
B 68.3/100LLM-oriented scraping API with page-to-markdown scrape, prompt-based JSON extraction, search, async crawl and change monitors.
Verdict An agent can mint an account and key over x402 or MPP with no signup. Rate limits and plan prices differ between docs pages.
Choose it for LLM pipelines that want prompt-to-JSON extraction and for agents that must pay with a wallet.
Strengths
- An agent can mint an account and key over x402 or MPP with no signup
- Per-call x402 on all five product endpoints through Orthogonal's gateway
- OpenAPI 3.1 with enums and documented 401, 402 and 429 responses
Weaknesses
- Rate limits and plan prices differ between docs pages
- Privacy policy from March 2024 with no retention periods, DPA or LLM provider list
- One account key with no scopes, and delete tools beside read tools on the MCP server
Price $20 / moAuth OAuth or keyx402 partialhosted
Olostep
B 63/100Scrape, crawl, map, batch and search API that returns markdown, HTML, JSON or screenshots, with JS rendering and residential IPs on every request.
Verdict OpenAPI 3.0.3 for every endpoint and a structured error body. The x402 challenge came back with no payment requirements in our probe.
Choose it for Bulk scraping on a budget and for batch jobs of thousands of URLs.
Strengths
- OpenAPI 3.0.3 for every endpoint and a structured error body
- 500 free successful requests with no card, $9 a month after that
- x402 documented on scrapes, maps, answers and crawls at $0.01 to $0.05
Weaknesses
- The x402 challenge came back with no payment requirements in our probe
- One dashboard key with no documented scopes, rotation or revocation
- No 429 or per-minute rate limits documented
Price $9 / moAuth API keyx402 partialhosted and local
Oxylabs Web Scraper API
B 63/100Hosted scraping API from Oxylabs that fetches public pages through its proxy pool with optional JavaScript rendering and returns HTML, Markdown or parsed JSON, with dedicated sources for Google, Amazon and other large sites.
Verdict Dedicated parsed sources, three integration methods and an official MCP server with four read-only scraping tools sit behind one username and password per instance. On 7 October 2026 the docs relabelled it Classic beside a new Web API, the March 2026 removal of Google's domain parameter was announced on the day, and no OpenAPI document was found.
Choose it for Agents that need parsed Google, Amazon and other large-site results or bulk asynchronous jobs through one API.
Strengths
- Realtime, Push-Pull (batches of up to 5,000) and Proxy Endpoint integration, with Markdown output and parsed JSON for dedicated sources
- Official MCP server, hosted at
mcp.oxylabs.ioor run locally, whose four scraping tools carryreadOnlyHintand enumerated inputs - Results with 5xx or 6xx codes and 429 responses are not billed, and faulted jobs can be retried at no cost
Weaknesses
- Google's
domainparameter was announced as gone on 18 March 2026 with no earlier changelog notice, and MCP v0.9.3 still exposes it - No OpenAPI document found. The contract is prose parameter tables and a Postman collection
- The docs relabelled the product Classic on 7 October 2026 and state no end date or migration plan
Price $30 / moAuth API keyx402 nohosted and local
Bright Data
C 59.8/100Web Unlocker, SERP API, a hosted Browser API and residential proxies, with an official MCP server wrapping them.
Verdict Pay only for successful requests, $1.50 per 1,000 on Web Unlocker and SERP. No public status page or incident history.
Choose it for Hard targets and site-specific structured data (Amazon, LinkedIn, Instagram, Google Maps) at volume.
Strengths
- Pay only for successful requests, $1.50 per 1,000 on Web Unlocker and SERP
- 5,000 free requests a month with no card
- Five tools by default out of 69, with groups for e-commerce, social, browser and more
Weaknesses
- No public status page or incident history
- The default hosted MCP URL carries the key as
?token= - No prompt-injection guidance for scraped content
Price $1.50 / 1k reqAuth OAuth or keyx402 nohosted
Head to head
- Apify MCP Server vs Firecrawl MCP BB 76.5 vs BB 75.3
- Apify MCP Server vs ZenRows BB 76.5 vs BB 74.8
- Apify MCP Server vs Spider BB 76.5 vs BB 74.1
- Apify MCP Server vs Zyte API BB 76.5 vs B 69.7
- Firecrawl MCP vs ZenRows BB 75.3 vs BB 74.8
- Firecrawl MCP vs Spider BB 75.3 vs BB 74.1
- Spider vs ZenRows BB 74.1 vs BB 74.8
- ZenRows vs Zyte API BB 74.8 vs B 69.7
- Spider vs Zyte API BB 74.1 vs B 69.7
Questions
What are the highest-rated web scraping and crawling APIs for AI agents?
Apify MCP Server has the highest benchmark score of the 18 ranked web scraping and crawling APIs, 76.5 (BB). Firecrawl MCP is second with 75.3 (BB).
How many web scraping and crawling APIs are agent-ready?
4 of the 18 ranked here grade BB or better, the bar for agent-ready on the Anchor benchmark.
Which web scraping and crawling APIs accept x402 payments?
Apify MCP Server and Spider. An agent can pay these per call in USDC with no account.
Which of these web scraping and crawling APIs is cheapest?
By published paid prices, Zyte API, at $0.06 per 1,000 requests, the lowest of the 14 listings here with a paid price in this unit (free allowances aside). Plans, volume tiers and free allowances change the sum, so check the listing's price table.
How is this list ranked?
By the Anchor benchmark score out of 100, a weighted mean of the scored categories minus deductions for negative events, from public evidence re-checked as vendors change. Listings cannot pay for a place. The latest assessment behind this page is from 9 October 2026.
How this list is made
The order is the Anchor benchmark score, the same number as on each listing and in the top list. Each listing is graded from public evidence against the benchmark checklist, and the picks above are worked out from those grades, prices and facts. No listing pays for its place, and paid audits or listing help never change a score.
Full ranked table · 167 head-to-head comparisons · Best tools in every category