# Research loop: find pages, then read the few that matter

> Research loop: find pages, then read the few that matter: 10 companies fully do it, 14 with a limit, 77 carry the tag. Every listing in Web scraping & crawling with the reason and a source.

- Canonical: https://www.anchorterminal.com/niches/web-scrapers.search-then-read/
- Markdown: https://www.anchorterminal.com/niches/web-scrapers.search-then-read/index.md (~2,200 tokens)
- Slim: https://www.anchorterminal.com/niches/web-scrapers.search-then-read/index.min.md (~430 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/niches/web-scrapers.search-then-read/index.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-15

Four or more companies' docs show a full fit.

| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |
| --- | --- | --- | --- | --- |
| 10 | 14 | 77 | 10 | 6 |

## What counts as done

A ranked list of URLs with snippets in one call, then each chosen page as Markdown under a set token budget, with no page costing more than its budget.

How agents ask for it: research this topic and cite sources; what is being said about X this week; find the docs for this error and read them

## Does the job

The docs show it meets the test above, and nothing we read says otherwise.

- **Apify MCP Server (Apify Technologies s.r.o.)**: apify--web-search returns titles, URLs and snippets and apify--web-fetch reads each page as Markdown at $0.001; both load by default and the README names this pairing for research prompts.
- **Bright Data (Bright Data Ltd.)**: search_engine returns JSON organic results (link, title, description) ten a page with country and page parameters, then scrape_as_markdown reads each at 1 credit; acc-21 pulls 100 results a call because SERP billing is per request.
- **Context.dev (Context Dev Inc.)**: Search returns up to 100 results for 1 credit per 10, markdownOptions scrapes each result to Markdown for 1 more credit with PDF handling and link retention. No token budget per page.
- **Crawl4AI (CONTEXT4AI PTE LTD)**: GET /search returns ranked results with title, URL and snippet for 1 credit, then /scrape reads each page at 0.5 credits; acc-08 and acc-09 run this loop. No token cap per page.
- **fastCRW (Nanofy OÜ)**: POST /v1/search returns up to 20 ranked results (default 5) for 1 credit, optional scraping of results adds 1 credit each, and maxLength bounds every page's characters.
- **Decodo Web Data API (UAB "Data troops")**: /search returns parsed Google organic results and top stories in under a second at $0.75 per 1,000 on the free plan, then /scrape with markdown: true reads each page.
- **Firecrawl MCP (SideGuide Technologies, Inc. (d/b/a Firecrawl))**: /search returns titles, descriptions, URLs and highlights at 2 credits per 10 results; scrapeOptions fetches each result as Markdown at 1 credit a page. Keyless MCP exposes search. acc-05 uses it for technical research. No token cap per page.
- **Olostep (Olostep Technologies Inc.)**: /v1/searches returns up to 25 links with title and description for 5 credits, and scrape_options embeds Markdown per link in the same call with a timeout; a slow link comes back null rather than failing the call.
- **Outcrawl (Aeonmind LLC (trading as Outcrawl))**: Search returns ranked results at 5 credits per 10, pro rata, and scrape: true renders each hit through the scrape pipeline at 1 credit in the same call; a search every engine refuses bills nothing.
- **Oxylabs Web Scraper API (oxylabs, UAB)**: POST /v1/search returns ranked results with max_results and geo parameters, then /v1/scrape with output markdown reads each page; the agent guide documents this search-then-scrape pattern.

## Does the job, with a limit

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

- **Diffbot (Diffbot Technologies Corp.)**: Web Search costs 1 credit a search and the product page says a response can carry a page's full content, but the /docs/search page returned 404 so the parameters and output shape weren't read.
- **Jina Reader (Jina AI GmbH)**: s.jina.ai returns the top five results with each page already read as Markdown, but keyless cannot search, every call costs at least 10,000 tokens, x-token-budget is ignored on search, and acc-31 saw single requests of 1.9M tokens.
- **Qovia (Qovia)**: POST /v1/search returns ranked items with text from ten Chinese social and news sources for $0.005 a call with paging, but there is no page fetch and it covers only the listed platforms, not the open web.
- **Scrape.do (PACKEND, LLC)**: Google Search API returns parsed SERP JSON at 10 credits a request, then each page can be fetched with output=markdown. No token or character cap on the page read.
- **ScrapeGraphAI (ScrapeGraphAI, Inc.)**: search returns up to 20 results with each page as Markdown in prune mode in one call at 2 credits a result. No token or character budget on the returned pages.
- **Scrapeless (NST LABS TECH LTD.)**: Google Search API returns parsed results by q, start and num; pages then go through the scrape job for markdown. No token cap, and the per-query price was not on the pages read.
- **ScrapeOps Proxy API Aggregator (ScrapeOps Limited)**: No search endpoint; a Google results page can be fetched and parsed with auto_extract, but the parser's cost isn't in the request-cost table and there is no token cap.
- **ScraperAPI (ScraperAPI LLC)**: Google SERP structured endpoint returns JSON (25 credits a request) and pages read with output_format=markdown; no token or character cap.
- **Scrapewise (BEBOTECH OÜ)**: google_search returns web, news or shopping results at 10 credits a query and fetch_page reads each page as text at 5 to 25 credits; no token cap.
- **ScrapingBee (VostokInc SAS)**: Google Search API returns organic_results and more at 10 credits (15 with light_request=false), then return_page_markdown reads each page. No token cap and multi-page billing is unstated.
- **Scrapingdog (OpenData Labs Private Limited)**: Google Search API returns parsed results at 5 to 10 credits, then /scrape with markdown=true reads pages; no token cap.
- **Spider (BAGELMEN LLC)**: /search returns ranked results and, with fetch_page_content, each page as Markdown in one call at bytes-plus-compute pricing. No token cap; search_limit bounds result count only.
- **WebScraping.AI (Urlooker LLC)**: /serp returns organic results with snippets at 15 credits a page and /text reads each page as Markdown; response_too_large asks you to fall back to /selected and there is no token cap.
- **Zyte API (Zyte Group Limited)**: The Search API returns typed organicResults for a query and domain, and markdown: true reads each page; no token cap.

## Couldn't check

The docs were closed to us, or silent. These carry no mark either way.

- **Microlink (Microlink Labs, S.L.)**: The MCP code registers a search tool and the listing carries search.serp, but no search page was found in the docs index read and the OpenAPI file has one path.

- not this job: BrowserAct (Hongkong Ziniao Technology Co., Limited): No search endpoint or tool; the MCP exposes bot and template search, task status and cancel only.
- not this job: jevcrawl (Rayen Mlayeh): No web search; crawl, scrape and map all start from a URL the agent already has.
- not this job: ScrapeAtlas (OD. Media & Technology LLC): No web search endpoint; keyword search exists only inside platforms (YouTube, Reddit, Quora) and returns records, not pages.
- not this job: Scrapfly (Joam Intelligence, LLC): No web search endpoint; the docs_search MCP tool searches Scrapfly's own docs.
- not this job: ScrapingAnt (DATAANT SP. Z O.O.): No search endpoint; Google pages come back as raw HTML at 10 credits through /v2/general.
- not this job: SoldFetch (SoldFetch): No web search; searches are eBay listing queries.
- not this job: ZenRows (ZENROWS, S.L.): No search endpoint among the five primitives.

1 listings in the category were added after this reading and aren't marked yet.

## What stops agents finishing

- search endpoint bills a fixed token minimum per call and spikes on large pages
- keyless tier cannot search
- search results return redirect URLs instead of destinations

## Around it

Before: search.serp. After: llm.synthesise, web-scrapers.page-to-markdown.

## Where this comes from

Weight 10 of 100 in Web scraping & crawling: 6 of 35 accounts; agents described as combining a search step with page reads (SearXNG plus Crawl4AI, Tavily plus Firecrawl, s.jina.ai)

As data: https://www.anchorterminal.com/api/v1/use-cases/web-scrapers.search-then-read.json, or the MCP tool get_use_case with id web-scrapers.search-then-read. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
