Niches · Web scraping & crawling
Research loop: find pages, then read the few that matter
Four or more companies' docs show a full fit.
What counts as done
A ranked list of URLs with snippets in one call, then each chosen page as Markdown under a set token budget, with no page costing more than its budget.
How agents ask for it: research this topic and cite sources
, what is being said about X this week
, find the docs for this error and read them
Does the job 10
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| Apify MCP Server A Apify Technologies s.r.o. | apify--web-search returns titles, URLs and snippets and apify--web-fetch reads each page as Markdown at $0.001; both load by default and the README names this pairing for research prompts. source |
| Bright Data C Bright Data Ltd. | search_engine returns JSON organic results (link, title, description) ten a page with country and page parameters, then scrape_as_markdown reads each at 1 credit; acc-21 pulls 100 results a call because SERP billing is per request. source |
| Context.dev A Context Dev Inc. | Search returns up to 100 results for 1 credit per 10, markdownOptions scrapes each result to Markdown for 1 more credit with PDF handling and link retention. No token budget per page. source |
| Crawl4AI D CONTEXT4AI PTE LTD | GET /search returns ranked results with title, URL and snippet for 1 credit, then /scrape reads each page at 0.5 credits; acc-08 and acc-09 run this loop. No token cap per page. source |
| fastCRW B Nanofy OÜ | POST /v1/search returns up to 20 ranked results (default 5) for 1 credit, optional scraping of results adds 1 credit each, and maxLength bounds every page's characters. source |
| Decodo Web Data API B UAB "Data troops" | /search returns parsed Google organic results and top stories in under a second at $0.75 per 1,000 on the free plan, then /scrape with markdown: true reads each page. source |
| Firecrawl MCP A SideGuide Technologies, Inc. (d/b/a Firecrawl) | /search returns titles, descriptions, URLs and highlights at 2 credits per 10 results; scrapeOptions fetches each result as Markdown at 1 credit a page. Keyless MCP exposes search. acc-05 uses it for technical research. No token cap per page. source |
| Olostep B Olostep Technologies Inc. | /v1/searches returns up to 25 links with title and description for 5 credits, and scrape_options embeds Markdown per link in the same call with a timeout; a slow link comes back null rather than failing the call. source |
| Outcrawl B Aeonmind LLC (trading as Outcrawl) | Search returns ranked results at 5 credits per 10, pro rata, and scrape: true renders each hit through the scrape pipeline at 1 credit in the same call; a search every engine refuses bills nothing. source |
| Oxylabs Web Scraper API B oxylabs, UAB | POST /v1/search returns ranked results with max_results and geo parameters, then /v1/scrape with output markdown reads each page; the agent guide documents this search-then-scrape pattern. source |
Does the job, with a limit 14
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Diffbot C Diffbot Technologies Corp. | Web Search costs 1 credit a search and the product page says a response can carry a page's full content, but the /docs/search page returned 404 so the parameters and output shape weren't read. source |
| Jina Reader C Jina AI GmbH | s.jina.ai returns the top five results with each page already read as Markdown, but keyless cannot search, every call costs at least 10,000 tokens, x-token-budget is ignored on search, and acc-31 saw single requests of 1.9M tokens. source |
| Qovia D Qovia | POST /v1/search returns ranked items with text from ten Chinese social and news sources for $0.005 a call with paging, but there is no page fetch and it covers only the listed platforms, not the open web. source |
| Scrape.do E PACKEND, LLC | Google Search API returns parsed SERP JSON at 10 credits a request, then each page can be fetched with output=markdown. No token or character cap on the page read. source |
| ScrapeGraphAI B ScrapeGraphAI, Inc. | search returns up to 20 results with each page as Markdown in prune mode in one call at 2 credits a result. No token or character budget on the returned pages. source |
| Scrapeless C NST LABS TECH LTD. | Google Search API returns parsed results by q, start and num; pages then go through the scrape job for markdown. No token cap, and the per-query price was not on the pages read. source |
| ScrapeOps Proxy API Aggregator D ScrapeOps Limited | No search endpoint; a Google results page can be fetched and parsed with auto_extract, but the parser's cost isn't in the request-cost table and there is no token cap. source |
| ScraperAPI D ScraperAPI LLC | Google SERP structured endpoint returns JSON (25 credits a request) and pages read with output_format=markdown; no token or character cap. source |
| Scrapewise B BEBOTECH OÜ | google_search returns web, news or shopping results at 10 credits a query and fetch_page reads each page as text at 5 to 25 credits; no token cap. source |
| ScrapingBee B VostokInc SAS | Google Search API returns organic_results and more at 10 credits (15 with light_request=false), then return_page_markdown reads each page. No token cap and multi-page billing is unstated. source |
| Scrapingdog E OpenData Labs Private Limited | Google Search API returns parsed results at 5 to 10 credits, then /scrape with markdown=true reads pages; no token cap. source |
| Spider BB BAGELMEN LLC | /search returns ranked results and, with fetch_page_content, each page as Markdown in one call at bytes-plus-compute pricing. No token cap; search_limit bounds result count only. source |
| WebScraping.AI B Urlooker LLC | /serp returns organic results with snippets at 15 credits a page and /text reads each page as Markdown; response_too_large asks you to fall back to /selected and there is no token cap. source |
| Zyte API BB Zyte Group Limited | The Search API returns typed organicResults for a query and domain, and markdown: true reads each page; no token cap. source |
Couldn't check 1
The docs were closed to us, or silent. These carry no mark either way.
| Listing | Why |
|---|---|
| Microlink B Microlink Labs, S.L. | The MCP code registers a search tool and the listing carries search.serp, but no search page was found in the docs index read and the OpenAPI file has one path. |
Doesn't do this job 7
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| BrowserAct D Hongkong Ziniao Technology Co., Limited | No search endpoint or tool; the MCP exposes bot and template search, task status and cancel only. source |
| jevcrawl B Rayen Mlayeh | No web search; crawl, scrape and map all start from a URL the agent already has. source |
| ScrapeAtlas B OD. Media & Technology LLC | No web search endpoint; keyword search exists only inside platforms (YouTube, Reddit, Quora) and returns records, not pages. source |
| Scrapfly BB Joam Intelligence, LLC | No web search endpoint; the docs_search MCP tool searches Scrapfly's own docs. source |
| ScrapingAnt E DATAANT SP. Z O.O. | No search endpoint; Google pages come back as raw HTML at 10 credits through /v2/general. source |
| SoldFetch E SoldFetch | No web search; searches are eBay listing queries. source |
| ZenRows A ZENROWS, S.L. | No search endpoint among the five primitives. source |
1 listings in the category were added after this reading and aren't marked yet.
What stops agents finishing
- search endpoint bills a fixed token minimum per call and spikes on large pages
- keyless tier cannot search
- search results return redirect URLs instead of destinations
Around it
Before: search.serp. After: llm.synthesise, web-scrapers.page-to-markdown.
Where this comes from
Weight 10 of 100 in Web scraping & crawling: 6 of 35 accounts; agents described as combining a search step with page reads (SearXNG plus Crawl4AI, Tavily plus Firecrawl, s.jina.ai)
The same as data: /api/v1/use-cases/web-scrapers.search-then-read.json, or the MCP tool get_use_case with id web-scrapers.search-then-read. Other jobs in this category: /api/v1/niches?category=web-scrapers.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.