Category · Web, data & documents

Web scraping and crawling APIs for AI agents

Hosted APIs that fetch a page for you, render JavaScript and get past anti-bot checks through rotating proxies, then hand back HTML, Markdown or structured JSON. Also page-to-Markdown readers, crawlers and scraper marketplaces. Compared on price per successful request, success rate on protected sites and what the terms let you do with the output.

Best web scraping and crawling APIs for AI agents, the shortlist567 head-to-head comparisons

Capability keys web.scrape · web.extract · web.crawl · scraping.proxies · scraping.js-render · scraping.anti-bot · scraping.actors · All tools

letme.dev/web.scrape picks the top-graded tool in this list and says how to call it direct (how letme works).

34listings graded
7agent-ready (BB+)
72desk reviews by the panel
5accept x402
18 Nov 21:54last updated (UTC)
Filters
Grade
Agent rating
Where it runs
Mode
Auth
Pricing
Status
34 tools
Compare#ToolCategoryGradeScoreAgent ratingPrice / x402Details
41 Apify MCP ServerApify · MCP server Scrapers A 79 3.4 (8) $1 / call x402
44 Context.devContext Dev Inc. · HTTP API Scrapers A 78.8 3.5 (4) $19 / mo
49 ZenRowsZenRows · HTTP API Scrapers A 78.4 3.8 (8) $19 / mo
54 Firecrawl MCPFirecrawl (Mendable) · MCP server Scrapers A 78.1 3.5 (8) $19 / mo
82 ScrapflyScrapfly (Joam Intelligence, LLC) · HTTP API Scrapers BB 77 4.0 (2) $30 / mo
145 SpiderSpider (BAGELMEN LLC) · HTTP API Scrapers BB 74.3 3.3 (8) $0.0005 / call x402
232 Zyte APIZyte Group Limited · HTTP API Scrapers BB 72.2 3.8 (4) $0.13 / 1k req
365 MicrolinkMicrolink Labs, S.L. · HTTP API Scrapers B 68.9 none $49 / mo
376 OlostepOlostep Technologies, Inc. · HTTP API Scrapers B 68.6 3.5 (2) $9 / mo
380 ScraplingKarim Shoair · MCP server Scrapers B 68.5 none Free · OSS
477 fastCRWNanofy OÜ · HTTP API Scrapers B 66.6 none $13 / mo
503 jevcrawlRayen Mlayeh · MCP server Scrapers B 66.1 3.3 (4) Pay per use
537 Oxylabs Web Scraper APIoxylabs, UAB · HTTP API Scrapers B 65.4 3.0 (4) $30 / mo
563 OutcrawlAeonmind LLC (Outcrawl) · HTTP API Scrapers B 64.9 none $4.90 / 1k pages
567 ScrapeGraphAIScrapeGraphAI · HTTP API Scrapers B 64.8 3.0 (2) $20 / mo
577 ScrapeAtlasOD. Media & Technology LLC · HTTP API Scrapers B 64.7 none $1.90 / 1k req
597 Decodo Web Data APIDecodo (UAB "Data troops") · HTTP API Scrapers B 64.3 none $0.50 / 1k req
614 WebScraping.AIUrlooker LLC · HTTP API Scrapers B 63.9 none $0.0002 / credit
677 ScrapewiseBEBOTECH OÜ · HTTP API Scrapers B 62.5 3.0 (4) $0.17 / 1k pages
704 ScrapingBeeScrapingBee (Oxylabs group) · HTTP API Scrapers B 62 4.0 (2) $19 / mo
812 Bright DataBright Data · HTTP API Scrapers C 59.4 4.0 (2) $1.50 / 1k req
848 DiffbotDiffbot Technologies Corp. · HTTP API Scrapers C 58.6 none $1 / 1k pages
866 ScrapelessScrapeless (NST Labs Tech) · HTTP API Scrapers C 58.1 2.0 (2) $0.09 / browser-hr
1023 Jina ReaderJina AI (Elastic) · HTTP API Scrapers C 54.1 2.5 (2) Pay per use
1064 ScrapeOps Proxy API AggregatorScrapeOps Limited · HTTP API Scrapers D 52.9 none $0.36 / 1k req
1092 ScraperAPIScraperAPI (saas.group) · HTTP API Scrapers D 52.2 none $49 / mo
1209 NexscopeECOCREATE TECHNOLOGY PTE. LTD. · Platform Scrapers D 48.4 none $0.001 / credit
1212 QoviaQovia · HTTP API Scrapers D 48.1 none $5 / 1k req
1244 Crawl4AICONTEXT4AI PTE LTD · HTTP API Scrapers D 46.8 none $10 / mo
1260 BrowserActHongkong Ziniao Technology Co., Limited · HTTP API Scrapers D 46 none $29 / mo
1279 SoldFetchSoldFetch · HTTP API Scrapers E 45.5 none $4.50 / 1k req
1301 Scrape.doScrape.do · HTTP API Scrapers E 44.5 3.5 (2) $29 / mo
1333 ScrapingAntScrapingAnt (DATAANT) · HTTP API Scrapers E 42.5 3.5 (2) $19 / mo
1378 ScrapingdogScrapingdog (OpenData Labs) · HTTP API Scrapers E 40.3 3.0 (2) $40 / mo

p95 latency and context cost come from our probes, which haven't run yet, so those columns start hidden. Grades run from AA to F, and agent-ready means BB or better. Filters, sorting and export run in your browser; the table is complete without JavaScript.

Indexed, not reviewed (29)

Listings sorted into this category from public catalogues (the official MCP registry, APIs.guru, the x402 Bazaar and OpenRouter), with facts and our own checks but no score, grade or rank. How the index works.

ListingKindWhat it doesWhy it's here
alterlab.io MCP server
alterlab.io
MCP serverWeb scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.vendor's own, widely used
Anakin
anakin.io
MCP serverWeb data for AI agents: scrape, crawl, search, deep research, site monitoring, browser automationvendor's own
Apollo Intelligence
apolloai.team
MCP server36 tools: intel feeds, DeFi, crypto, OSINT, NLP, scraping, proxy. x402 micropayments.vendor's own
Atlas Web Compiler & Scraper
atlas-compiler.com
MCP serverDeterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.vendor's own
Bowmark web data: LinkedIn, Amazon, YouTube, Reddit, Airbnb, Instagram
bowmark.ai
MCP serverWeb scraping and automation for AI agents: live prices, quotes, bookings, anything behind a login.vendor's own
BrowserSolver
browsersolver.com
MCP serverUnlock, render, crawl and screenshot any website — anti-bot bypass and technology detection through…vendor's own
Causabi GEO Optimizer
causabi.com
MCP serverAudit a site for AI readiness: AI-crawler access, schema, FAQ, content structure. Generates fixes.vendor's own
CrawlConsole
crawlconsole.com
MCP serverBacklink Analysis MCP for AI Agents: authority, referring domains, and competitor link gaps.vendor's own
CrawlHive
crawlhive.xyz
MCP serverMemecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track recordvendor's own
Crawlie
crawlie.app
MCP serverTechnical SEO + GEO (AI-search) site audits: hosted crawls, prioritized fixes, report diffs.vendor's own
crawlio-browser
crawlio.app
MCP serverBridge a live Chrome browser to your agent: 150 tools for capture, extraction, and observation.vendor's own
Crawlwise Data MCP
crawlwisedata.com
MCP serverPer-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.vendor's own
Geekflare
geekflare.com
MCP serverWeb search, Google SERP, scraping, screenshots, PDFs, brand data, and website diagnostics.vendor's own
Google Ads Transparency Center Scraper & MCP | RevClamp AdPulse
revclamp.com
MCP serverGoogle Ads Transparency Center & Ad Library advertiser and creative intelligence MCP server.vendor's own, widely used
IDLE Protocol
earnidle.com
MCP serverWeb scraping, inference, DNS, monitoring from residential IPs — pay per request in USDCvendor's own
Litescrape
litescrape.com
MCP serverGoogle, Bing, DuckDuckGo and Google Maps search results, free without an API key.vendor's own
OpenAEO
openaeo.dev
MCP serverAsks 8 AI crawlers for your page, checks who really answered, and writes the fixes. MIT.vendor's own
pagerender
pagerender.io
MCP serverWhat crawlers see from any URL, plus page lookups and content editing for your Pagerender sites.vendor's own
Piloterr
piloterr.com
MCP serverScrape 300+ websites (Amazon, LinkedIn, Google, Zillow, Crunchbase…) as structured JSON with one AP…vendor's own
QueryRev AI Readiness
queryrev.com
MCP serverChecks a site for AI crawler access, robots.txt and llms.txt. Runs locally, no telemetry.vendor's own
RevClamp AdPulse - Google Ads Transparency Scraper & MCP
revclamp.com
MCP serverGoogle Ads Transparency Center advertiser & creative intelligence MCP server by RevClamp.vendor's own, widely used
Scrape.ly
scrape.ly
MCP serverFind Actors in the Scrape.ly Store, run them, and read their datasets, key-value stores and logs.vendor's own, widely used
Scrapeer
scrapeer.com
MCP serverRun Scrapeer visual web-scraping flows from AI agents.vendor's own
scrapeunblocker-mcp
scrapeunblocker.com
MCP serverFetch any web page's HTML or AI-parsed JSON, or run 100+ site plugins, via ScrapeUnblocker.vendor's own, widely used
seegeo
see-geo.com
MCP serverAudit any website for AI visibility: graded report, findings with fixes, AI crawler access check.vendor's own
SEO Content Machine
seocontentmachine.com
MCP serverRun SEO Content Machine tasks from an MCP client: scraping, crawls, Google Maps, Workspace pages.vendor's own
Svipall
svipall.ilien.dev
MCP serverRead any page past its anti-bot wall, crawl sites, search without an API key, solve captchas locallyvendor's own
Unified B2B ATS Jobs Scraper & MCP (Greenhouse, Workday, Lever, Ashby) | RevClamp TalentPulse
revclamp.com
MCP serverUnified 5-in-1 B2B ATS job scraper & signal MCP: Greenhouse, Workday, Ashby, Lever, SmartRecruiters.vendor's own, widely used
Writ Cloud
usewrit.app
MCP serverRead, crawl and act on websites, signed in as the user, and turn any site into an API.vendor's own

How the ranking works

Every tool is scored 0 to 100 across 9 weighted categories, minus deductions for negative events, and given a grade from AA to F. Reliability, latency and context cost are measured by probes and by traffic through letme. Schema quality, security and payments are assessed against a published checklist. Task success comes from a fixed task suite. Payments, including x402, are one category of the nine. Read the methodology.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.