# Web scraping and crawling APIs for AI agents

> 14 web scraping & crawling listings ranked by the Anchor benchmark. Leader Apify MCP Server (A). Hosted APIs that fetch a page for you, render JavaScript and get past anti-bot checks through rotating proxies, then hand back HTML, Markdown or structured JSON. Also page-to-Markdown readers, crawlers and scraper marketplaces. Compared on price per successful request, success rate on protected sites and what the terms let you do with the output.

- Canonical: https://www.anchorterminal.com/categories/web-scrapers
- Markdown: https://www.anchorterminal.com/categories/web-scrapers.md (~4,750 tokens)
- Slim: https://www.anchorterminal.com/categories/web-scrapers.min.md (~680 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/categories/web-scrapers.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (full text: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- Updated: 2026-10-04

Hosted APIs that fetch a page for you, render JavaScript and get past anti-bot checks through rotating proxies, then hand back HTML, Markdown or structured JSON. Also page-to-Markdown readers, crawlers and scraper marketplaces. Compared on price per successful request, success rate on protected sites and what the terms let you do with the output.

- Tools ranked: 14 · agent-ready (BB or better): 4 · accept x402: 5 · hosted endpoints: 14 · desk reviews by the panel: 52
- JSON: https://www.anchorterminal.com/api/v1/tools.json (list) · https://www.anchorterminal.com/api/v1/rankings.json (ranked) · https://www.anchorterminal.com/api/v1/x402.json (payable) · https://www.anchorterminal.com/api/v1/capabilities.json (by capability)
- Grades run AA, A, BB, B, C, D, E, F · methodology: https://www.anchorterminal.com/benchmark/

- Capabilities in this category: web.scrape, web.extract, web.crawl, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.actors
- https://letme.dev/web.scrape picks the top-graded tool in this list and says how to call it direct; calling through letme comes later (https://www.anchorterminal.com/letme/index.md)

## Ranking

| # | Tool | Vendor | Kind | Category | Grade | Score | Confidence | x402 | Auth | Where | Reviews | Page |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 11 | Apify MCP Server | Apify | MCP server | Scrapers | A | 78.6 | medium | yes ($1/call) | OAuth or key | hosted + local | 3.4/5 (8) | https://www.anchorterminal.com/tools/apify-mcp.md |
| 34 | Firecrawl MCP | Firecrawl (Mendable) | MCP server | Scrapers | BB | 75.5 | high | no | OAuth or key | hosted + local | 3.5/5 (8) | https://www.anchorterminal.com/tools/firecrawl-mcp.md |
| 43 | ZenRows | ZenRows | HTTP API | Scrapers | BB | 75.1 | medium | payer | OAuth or key | hosted + local | 3.8/5 (8) | https://www.anchorterminal.com/tools/zenrows.md |
| 49 | Spider | Spider (BAGELMEN LLC) | HTTP API | Scrapers | BB | 74.3 | medium | yes ($0.0005/call) | OAuth or key | hosted + local | 3.3/5 (8) | https://www.anchorterminal.com/tools/spider-cloud.md |
| 107 | Scrapfly | Scrapfly (Joam Intelligence, LLC) | HTTP API | Scrapers | B | 69.8 | medium | no | OAuth or key | hosted + local | 4/5 (2) | https://www.anchorterminal.com/tools/scrapfly.md |
| 128 | ScrapeGraphAI | ScrapeGraphAI | HTTP API | Scrapers | B | 68.3 | medium | payer | OAuth or key | hosted | 3/5 (2) | https://www.anchorterminal.com/tools/scrapegraphai.md |
| 210 | Olostep | Olostep Technologies, Inc. | HTTP API | Scrapers | B | 63.1 | medium | payer | API key | hosted + local | 3.5/5 (2) | https://www.anchorterminal.com/tools/olostep.md |
| 254 | Bright Data | Bright Data | HTTP API | Scrapers | C | 60.1 | medium | no | OAuth or key | hosted | 4/5 (2) | https://www.anchorterminal.com/tools/bright-data.md |
| 268 | ScrapingBee | ScrapingBee (Oxylabs group) | HTTP API | Scrapers | C | 59.3 | medium | no | API key | hosted | 4/5 (2) | https://www.anchorterminal.com/tools/scrapingbee.md |
| 326 | Scrapeless | Scrapeless (NST Labs Tech) | HTTP API | Scrapers | C | 54.3 | medium | no | API key | hosted + local | 2/5 (2) | https://www.anchorterminal.com/tools/scrapeless.md |
| 402 | Jina Reader | Jina AI (Elastic) | HTTP API | Scrapers | E | 45.3 | medium | no | OAuth or key | hosted | 2.5/5 (2) | https://www.anchorterminal.com/tools/jina-reader.md |
| 406 | Scrape.do | Scrape.do | HTTP API | Scrapers | E | 44 | medium | no | API key | hosted | 3.5/5 (2) | https://www.anchorterminal.com/tools/scrape-do.md |
| 425 | Scrapingdog | Scrapingdog (OpenData Labs) | HTTP API | Scrapers | E | 39.3 | medium | no | API key | hosted + local | 3/5 (2) | https://www.anchorterminal.com/tools/scrapingdog.md |
| 440 | ScrapingAnt | ScrapingAnt (DATAANT) | HTTP API | Scrapers | F | 35.9 | medium | no | API key | hosted | 3.5/5 (2) | https://www.anchorterminal.com/tools/scrapingant.md |

Scores are from public evidence against the published checklist (https://www.anchorterminal.com/benchmark/), with Performance and Task success pending. p95 latency and context cost come from our probes, which haven't run yet.

## Summaries

### 11. Apify MCP Server, A (78.6)

Exposes thousands of Apify Store Actors (scrapers, crawlers, automations) as dynamically discovered MCP tools, hosted at mcp.apify.com or run locally; supports OAuth, API tokens and agentic payments (x402 prepaid tokens, Skyfire). x402 on Apify's own domains, a prepaid token from agi.apify.com for any Actor and per-run payment on mcp.apify.com for Pay Per Event Actors. API and Actor runs timed out for about 12 hours on 21 and 22 July 2026.

- Page: https://www.anchorterminal.com/tools/apify-mcp · Markdown: https://www.anchorterminal.com/tools/apify-mcp.md · JSON: https://www.anchorterminal.com/api/v1/tools/apify-mcp.json
- Capabilities: scraping.actors, web.crawl · endpoint: `https://mcp.apify.com`

### 34. Firecrawl MCP, BB (75.5)

Official Firecrawl server for scraping pages to markdown/JSON, mapping and crawling sites, web search, browser interaction, monitoring and an autonomous research agent. Keyless hosted endpoint with scrape, search and parse, plus OAuth and a search-only endpoint. No x402 or other machine payment.

- Page: https://www.anchorterminal.com/tools/firecrawl-mcp · Markdown: https://www.anchorterminal.com/tools/firecrawl-mcp.md · JSON: https://www.anchorterminal.com/api/v1/tools/firecrawl-mcp.json
- Capabilities: web.search, web.fetch, web.crawl, web.extract · endpoint: `https://mcp.firecrawl.dev/v2/mcp`

### 43. ZenRows, BB (75.1)

Scraping API (Fetch, formerly Universal Scraper API) with JavaScript rendering, residential proxies and anti-bot bypass, plus Extract for structured JSON, Batch for bulk URL jobs and hosted Browser Sessions. No incidents from July to 1 October 2026 on a six-component Better Stack status page. The Fetch API only takes the key in the query string.

- Page: https://www.anchorterminal.com/tools/zenrows · Markdown: https://www.anchorterminal.com/tools/zenrows.md · JSON: https://www.anchorterminal.com/api/v1/tools/zenrows.json
- Capabilities: web.scrape, web.extract, browser.hosted, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot · endpoint: `https://api.zenrows.com/v1/`

### 49. Spider, BB (74.3)

Hosted crawl, scrape, search, links and screenshot API built on the open-source Rust spider crate, plus a cloud browser and proxy pool. Keyless /scrape and working x402 v2 on every core route, so an agent needs no account. Operated by BAGELMEN LLC with no address, retention periods or DPA in the privacy policy.

- Page: https://www.anchorterminal.com/tools/spider-cloud · Markdown: https://www.anchorterminal.com/tools/spider-cloud.md · JSON: https://www.anchorterminal.com/api/v1/tools/spider-cloud.json
- Capabilities: web.scrape, web.extract, web.crawl, web.search, web.fetch, browser.hosted, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot · endpoint: `https://api.spider.cloud`

### 107. Scrapfly, B (69.8)

Scrape API with datacenter or residential proxies, JS rendering and an anti-bot Unblocker, plus screenshot, LLM extraction, crawler and hosted Cloud Browser APIs on one credit balance. The error catalogue documents HTTP status, retry behaviour and billing for over 100 codes. The MCP server loads 61 tools at once.

- Page: https://www.anchorterminal.com/tools/scrapfly · Markdown: https://www.anchorterminal.com/tools/scrapfly.md · JSON: https://www.anchorterminal.com/api/v1/tools/scrapfly.json
- Capabilities: web.scrape, web.fetch, web.extract, web.crawl, browser.hosted, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot · endpoint: `https://api.scrapfly.io/scrape`

### 128. ScrapeGraphAI, B (68.3)

LLM-oriented scraping API with page-to-markdown scrape, prompt-based JSON extraction, search, async crawl and change monitors. An agent can mint an account and key over x402 or MPP with no signup. Rate limits and plan prices differ between docs pages.

- Page: https://www.anchorterminal.com/tools/scrapegraphai · Markdown: https://www.anchorterminal.com/tools/scrapegraphai.md · JSON: https://www.anchorterminal.com/api/v1/tools/scrapegraphai.json
- Capabilities: web.scrape, web.extract, web.crawl, web.search, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot · endpoint: `https://v2-api.scrapegraphai.com`

### 210. Olostep, B (63.1)

Scrape, crawl, map, batch and search API that returns markdown, HTML, JSON or screenshots, with JS rendering and residential IPs on every request. OpenAPI 3.0.3 for every endpoint and a structured error body. The x402 challenge came back with no payment requirements in our probe.

- Page: https://www.anchorterminal.com/tools/olostep · Markdown: https://www.anchorterminal.com/tools/olostep.md · JSON: https://www.anchorterminal.com/api/v1/tools/olostep.json
- Capabilities: web.scrape, web.extract, web.crawl, web.search, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot, scraping.serp · endpoint: `https://api.olostep.com`

### 254. Bright Data, C (60.1)

Web Unlocker, SERP API, a hosted Browser API and residential proxies, with an official MCP server wrapping them. Pay only for successful requests, $1.50 per 1,000 on Web Unlocker and SERP. No public status page or incident history.

- Page: https://www.anchorterminal.com/tools/bright-data · Markdown: https://www.anchorterminal.com/tools/bright-data.md · JSON: https://www.anchorterminal.com/api/v1/tools/bright-data.json
- Capabilities: web.scrape, web.search, browser.hosted, web.crawl · endpoint: `https://api.brightdata.com/request`

### 268. ScrapingBee, C (59.3)

HTML API that fetches a URL through rotating, premium or stealth proxies with headless rendering, JavaScript scenarios, screenshots and CSS or AI extraction. Dedicated Google, Amazon, Walmart, YouTube, ChatGPT and Gemini endpoints next to the general HTML API. remote-mcp.txt says the hosted MCP only takes the key as ?api_key= in its URL.

- Page: https://www.anchorterminal.com/tools/scrapingbee · Markdown: https://www.anchorterminal.com/tools/scrapingbee.md · JSON: https://www.anchorterminal.com/api/v1/tools/scrapingbee.json
- Capabilities: web.scrape, web.extract, web.search, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot, scraping.serp · endpoint: `https://app.scrapingbee.com/api/v1/`

### 326. Scrapeless, C (54.3)

Web Unlocker, a hosted cloud browser over CDP, a crawl API, Google and Amazon scraping APIs, AI answer-engine scrapers and residential proxies under one key. Hosted streamable HTTP MCP at api.scrapeless.com/mcp with header auth, plus a stdio package, both in the official MCP registry. llms.txt quotes Deep SerpApi at $0.1 per 1,000 while the plan data charges $1 per 1,000 for Google Search.

- Page: https://www.anchorterminal.com/tools/scrapeless · Markdown: https://www.anchorterminal.com/tools/scrapeless.md · JSON: https://www.anchorterminal.com/api/v1/tools/scrapeless.json
- Capabilities: web.scrape, web.crawl, web.search, browser.hosted, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot, scraping.serp · endpoint: `https://api.scrapeless.com`

### 402. Jina Reader, E (45.3)

API that converts web pages and search results to Markdown for agents and applications. One URL prefix, no key needed, Markdown back. No price per token in any currency on the public page.

- Page: https://www.anchorterminal.com/tools/jina-reader · Markdown: https://www.anchorterminal.com/tools/jina-reader.md · JSON: https://www.anchorterminal.com/api/v1/tools/jina-reader.json
- Capabilities: web.scrape, web.fetch, web.search · endpoint: `https://r.jina.ai/`

### 406. Scrape.do, E (44)

Single GET endpoint that fetches a URL through rotating datacenter, residential or mobile proxies, with optional headless rendering, screenshots, markdown output and a proxy mode. $29 for 250,000 basic credits, and 429s, 502s and most failures aren't billed. Token travels in the query string on the main API.

- Page: https://www.anchorterminal.com/tools/scrape-do · Markdown: https://www.anchorterminal.com/tools/scrape-do.md · JSON: https://www.anchorterminal.com/api/v1/tools/scrape-do.json
- Capabilities: web.scrape, web.fetch, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot, scraping.serp · endpoint: `https://api.scrape.do/`

### 425. Scrapingdog, E (39.3)

Web scraping API billed by credits. A plain request is 1 credit, about $0.20 per 1,000 on Lite. API key only travels in the URL query string.

- Page: https://www.anchorterminal.com/tools/scrapingdog · Markdown: https://www.anchorterminal.com/tools/scrapingdog.md · JSON: https://www.anchorterminal.com/api/v1/tools/scrapingdog.json
- Capabilities: web.scrape, web.extract, web.search, scraping.proxies, scraping.js-render, scraping.anti-bot, scraping.screenshot, scraping.serp · endpoint: `https://api.scrapingdog.com/scrape`

### 440. ScrapingAnt, F (35.9)

Scraping API that runs each request in headless Chrome behind rotating datacenter or residential proxies, with markdown and AI extraction endpoints, a 3-tool hosted MCP server and standalone proxies billed by the GB. 10,000 free credits a month with no card. API Basics puts the key in the query string, so it lands in logs.

- Page: https://www.anchorterminal.com/tools/scrapingant · Markdown: https://www.anchorterminal.com/tools/scrapingant.md · JSON: https://www.anchorterminal.com/api/v1/tools/scrapingant.json
- Capabilities: web.scrape, web.fetch, web.extract, scraping.proxies, scraping.js-render, scraping.anti-bot · endpoint: `https://api.scrapingant.com/v2/general`

## Indexed, not reviewed (22)

Sorted into this category from public catalogues, with facts and our own checks but no score, grade or rank (https://www.anchorterminal.com/indexed/index.md).

| Listing | Kind | What it does | Why it's here |
| --- | --- | --- | --- |
| [alterlab.io MCP server](https://www.anchorterminal.com/tools/alterlab-mcp-server.md) | MCP server | Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass. | vendor's own |
| [Anakin](https://www.anchorterminal.com/tools/anakin-mcp.md) | MCP server | Web data for AI agents: scrape, crawl, search, deep research, site monitoring, browser automation | vendor's own |
| [Apollo Intelligence](https://www.anchorterminal.com/tools/apolloai-intelligence.md) | MCP server | 36 tools: intel feeds, DeFi, crypto, OSINT, NLP, scraping, proxy. x402 micropayments. | vendor's own |
| [CrawlConsole](https://www.anchorterminal.com/tools/crawlconsole.md) | MCP server | Backlink Analysis MCP for AI Agents: authority, referring domains, and competitor link gaps. | vendor's own |
| [Crawlie](https://www.anchorterminal.com/tools/crawlie.md) | MCP server | Technical SEO + GEO (AI-search) site audits: hosted crawls, prioritized fixes, report diffs. | vendor's own |
| [crawlio-browser](https://www.anchorterminal.com/tools/crawlio-browser.md) | MCP server | Bridge a live Chrome browser to your agent: 150 tools for capture, extraction, and observation. | vendor's own |
| [geekflare.com MCP server](https://www.anchorterminal.com/tools/geekflare-mcp.md) | MCP server | Geekflare MCP server for scraping, search, screenshots, broken links, DNS lookup, and network tools. | vendor's own |
| [Glasser](https://www.anchorterminal.com/tools/glasser.md) | MCP server | One key to 1,000+ paid data APIs: enrichment, SEO/SERP, scraping, places, news. Pay per call. | vendor's own |
| [glim.sh](https://www.anchorterminal.com/tools/glim.md) | MCP server | Give your agent live data from Twitter, Reddit, the web and GitHub. No API keys, no scraping stack. | vendor's own |
| [Google Maps Scraper](https://www.anchorterminal.com/tools/gmapsextractor-google-maps-scraper.md) | MCP server | Live Google Maps business search, review, and photo data for AI agents over MCP. | vendor's own |
| [IDLE Protocol](https://www.anchorterminal.com/tools/earnidle-mcp.md) | MCP server | Web scraping, inference, DNS, monitoring from residential IPs — pay per request in USDC | vendor's own |
| [Litescrape](https://www.anchorterminal.com/tools/litescrape-mcp-server.md) | MCP server | Google, Bing, DuckDuckGo and Google Maps search results, free without an API key. | vendor's own |
| [mcp-gera-jobs](https://www.anchorterminal.com/tools/mcp-gera-jobs.md) | MCP server | Search global job categories, real crawled salary benchmarks and country job markets. No auth. | vendor's own |
| [OpenAEO](https://www.anchorterminal.com/tools/openaeo-audit.md) | MCP server | Asks 8 AI crawlers for your page, checks who really answered, and writes the fixes. MIT. | vendor's own |
| [oxylabs-mcp](https://www.anchorterminal.com/tools/oxylabs-mcp.md) | MCP server | Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API. | vendor's own |
| [Rampify](https://www.anchorterminal.com/tools/rampify-mcp.md) | MCP server | SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent. | vendor's own |
| [ReplyNodes](https://www.anchorterminal.com/tools/replynodes-mcp.md) | MCP server | Web MCP: scrape/crawl sites, web search, brand assets, app stores, YouTube, Reddit, Hacker News. | vendor's own |
| [Scrapeer](https://www.anchorterminal.com/tools/scrapeer-mcp-server.md) | MCP server | Run Scrapeer visual web-scraping flows from AI agents. | vendor's own |
| [scrapeunblocker-mcp](https://www.anchorterminal.com/tools/scrapeunblocker-mcp.md) | MCP server | Fetch any web page's HTML or AI-parsed JSON through the ScrapeUnblocker anti-bot API. | vendor's own |
| [seegeo](https://www.anchorterminal.com/tools/see-geo-seegeo.md) | MCP server | Audit any website for AI visibility: graded report, findings with fixes, AI crawler access check. | vendor's own |
| [Svipall](https://www.anchorterminal.com/tools/ilien-mcp.md) | MCP server | Read any page past its anti-bot wall, crawl sites, search without an API key, solve captchas locally | vendor's own |
| [tooltrace.io MCP server](https://www.anchorterminal.com/tools/tooltrace-mcp-server.md) | MCP server | Web tools for AI agents: scrape pages to Markdown, audit SEO, detect tech stacks, check sitemaps | vendor's own |