Niches · Web scraping & crawling

Research loop: find pages, then read the few that matter

Four or more companies' docs show a full fit.

10companies, full fit
14companies, fit with a limit
77companies carry the tag
10share of the category's work
6first-hand accounts

What counts as done

A ranked list of URLs with snippets in one call, then each chosen page as Markdown under a set token budget, with no page costing more than its budget.

How agents ask for it: research this topic and cite sources, what is being said about X this week, find the docs for this error and read them

Does the job 10

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
Apify MCP Server A
Apify Technologies s.r.o.
apify--web-search returns titles, URLs and snippets and apify--web-fetch reads each page as Markdown at $0.001; both load by default and the README names this pairing for research prompts. source
Bright Data C
Bright Data Ltd.
search_engine returns JSON organic results (link, title, description) ten a page with country and page parameters, then scrape_as_markdown reads each at 1 credit; acc-21 pulls 100 results a call because SERP billing is per request. source
Context.dev A
Context Dev Inc.
Search returns up to 100 results for 1 credit per 10, markdownOptions scrapes each result to Markdown for 1 more credit with PDF handling and link retention. No token budget per page. source
Crawl4AI D
CONTEXT4AI PTE LTD
GET /search returns ranked results with title, URL and snippet for 1 credit, then /scrape reads each page at 0.5 credits; acc-08 and acc-09 run this loop. No token cap per page. source
fastCRW B
Nanofy OÜ
POST /v1/search returns up to 20 ranked results (default 5) for 1 credit, optional scraping of results adds 1 credit each, and maxLength bounds every page's characters. source
Decodo Web Data API B
UAB "Data troops"
/search returns parsed Google organic results and top stories in under a second at $0.75 per 1,000 on the free plan, then /scrape with markdown: true reads each page. source
Firecrawl MCP A
SideGuide Technologies, Inc. (d/b/a Firecrawl)
/search returns titles, descriptions, URLs and highlights at 2 credits per 10 results; scrapeOptions fetches each result as Markdown at 1 credit a page. Keyless MCP exposes search. acc-05 uses it for technical research. No token cap per page. source
Olostep B
Olostep Technologies Inc.
/v1/searches returns up to 25 links with title and description for 5 credits, and scrape_options embeds Markdown per link in the same call with a timeout; a slow link comes back null rather than failing the call. source
Outcrawl B
Aeonmind LLC (trading as Outcrawl)
Search returns ranked results at 5 credits per 10, pro rata, and scrape: true renders each hit through the scrape pipeline at 1 credit in the same call; a search every engine refuses bills nothing. source
Oxylabs Web Scraper API B
oxylabs, UAB
POST /v1/search returns ranked results with max_results and geo parameters, then /v1/scrape with output markdown reads each page; the agent guide documents this search-then-scrape pattern. source

Does the job, with a limit 14

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Diffbot C
Diffbot Technologies Corp.
Web Search costs 1 credit a search and the product page says a response can carry a page's full content, but the /docs/search page returned 404 so the parameters and output shape weren't read. source
Jina Reader C
Jina AI GmbH
s.jina.ai returns the top five results with each page already read as Markdown, but keyless cannot search, every call costs at least 10,000 tokens, x-token-budget is ignored on search, and acc-31 saw single requests of 1.9M tokens. source
Qovia D
Qovia
POST /v1/search returns ranked items with text from ten Chinese social and news sources for $0.005 a call with paging, but there is no page fetch and it covers only the listed platforms, not the open web. source
Scrape.do E
PACKEND, LLC
Google Search API returns parsed SERP JSON at 10 credits a request, then each page can be fetched with output=markdown. No token or character cap on the page read. source
ScrapeGraphAI B
ScrapeGraphAI, Inc.
search returns up to 20 results with each page as Markdown in prune mode in one call at 2 credits a result. No token or character budget on the returned pages. source
Scrapeless C
NST LABS TECH LTD.
Google Search API returns parsed results by q, start and num; pages then go through the scrape job for markdown. No token cap, and the per-query price was not on the pages read. source
ScrapeOps Proxy API Aggregator D
ScrapeOps Limited
No search endpoint; a Google results page can be fetched and parsed with auto_extract, but the parser's cost isn't in the request-cost table and there is no token cap. source
ScraperAPI D
ScraperAPI LLC
Google SERP structured endpoint returns JSON (25 credits a request) and pages read with output_format=markdown; no token or character cap. source
Scrapewise B
BEBOTECH OÜ
google_search returns web, news or shopping results at 10 credits a query and fetch_page reads each page as text at 5 to 25 credits; no token cap. source
ScrapingBee B
VostokInc SAS
Google Search API returns organic_results and more at 10 credits (15 with light_request=false), then return_page_markdown reads each page. No token cap and multi-page billing is unstated. source
Scrapingdog E
OpenData Labs Private Limited
Google Search API returns parsed results at 5 to 10 credits, then /scrape with markdown=true reads pages; no token cap. source
Spider BB
BAGELMEN LLC
/search returns ranked results and, with fetch_page_content, each page as Markdown in one call at bytes-plus-compute pricing. No token cap; search_limit bounds result count only. source
WebScraping.AI B
Urlooker LLC
/serp returns organic results with snippets at 15 credits a page and /text reads each page as Markdown; response_too_large asks you to fall back to /selected and there is no token cap. source
Zyte API BB
Zyte Group Limited
The Search API returns typed organicResults for a query and domain, and markdown: true reads each page; no token cap. source

Couldn't check 1

The docs were closed to us, or silent. These carry no mark either way.

ListingWhy
Microlink B
Microlink Labs, S.L.
The MCP code registers a search tool and the listing carries search.serp, but no search page was found in the docs index read and the OpenAPI file has one path.

Doesn't do this job 7

Listings in the category that do something else, with the reason.

ListingWhy
BrowserAct D
Hongkong Ziniao Technology Co., Limited
No search endpoint or tool; the MCP exposes bot and template search, task status and cancel only. source
jevcrawl B
Rayen Mlayeh
No web search; crawl, scrape and map all start from a URL the agent already has. source
ScrapeAtlas B
OD. Media & Technology LLC
No web search endpoint; keyword search exists only inside platforms (YouTube, Reddit, Quora) and returns records, not pages. source
Scrapfly BB
Joam Intelligence, LLC
No web search endpoint; the docs_search MCP tool searches Scrapfly's own docs. source
ScrapingAnt E
DATAANT SP. Z O.O.
No search endpoint; Google pages come back as raw HTML at 10 credits through /v2/general. source
SoldFetch E
SoldFetch
No web search; searches are eBay listing queries. source
ZenRows A
ZENROWS, S.L.
No search endpoint among the five primitives. source

1 listings in the category were added after this reading and aren't marked yet.

What stops agents finishing

  • search endpoint bills a fixed token minimum per call and spikes on large pages
  • keyless tier cannot search
  • search results return redirect URLs instead of destinations

Around it

Before: search.serp. After: llm.synthesise, web-scrapers.page-to-markdown.

Where this comes from

Weight 10 of 100 in Web scraping & crawling: 6 of 35 accounts; agents described as combining a search step with page reads (SearXNG plus Crawl4AI, Tavily plus Firecrawl, s.jina.ai)

The same as data: /api/v1/use-cases/web-scrapers.search-then-read.json, or the MCP tool get_use_case with id web-scrapers.search-then-read. Other jobs in this category: /api/v1/niches?category=web-scrapers.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.