{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/web-scrapers.search-then-read/",
    "json": "https://www.anchorterminal.com/niches/web-scrapers.search-then-read/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/web-scrapers.search-then-read/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/web-scrapers.search-then-read/index.min.md"
  },
  "markdown": "Four or more companies' docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 10 | 14 | 77 | 10 | 6 |\n\n## What counts as done\n\nA ranked list of URLs with snippets in one call, then each chosen page as Markdown under a set token budget, with no page costing more than its budget.\n\nHow agents ask for it: research this topic and cite sources; what is being said about X this week; find the docs for this error and read them\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **Apify MCP Server (Apify Technologies s.r.o.)**: apify--web-search returns titles, URLs and snippets and apify--web-fetch reads each page as Markdown at $0.001; both load by default and the README names this pairing for research prompts.\n- **Bright Data (Bright Data Ltd.)**: search_engine returns JSON organic results (link, title, description) ten a page with country and page parameters, then scrape_as_markdown reads each at 1 credit; acc-21 pulls 100 results a call because SERP billing is per request.\n- **Context.dev (Context Dev Inc.)**: Search returns up to 100 results for 1 credit per 10, markdownOptions scrapes each result to Markdown for 1 more credit with PDF handling and link retention. No token budget per page.\n- **Crawl4AI (CONTEXT4AI PTE LTD)**: GET /search returns ranked results with title, URL and snippet for 1 credit, then /scrape reads each page at 0.5 credits; acc-08 and acc-09 run this loop. No token cap per page.\n- **fastCRW (Nanofy OÜ)**: POST /v1/search returns up to 20 ranked results (default 5) for 1 credit, optional scraping of results adds 1 credit each, and maxLength bounds every page's characters.\n- **Decodo Web Data API (UAB \"Data troops\")**: /search returns parsed Google organic results and top stories in under a second at $0.75 per 1,000 on the free plan, then /scrape with markdown: true reads each page.\n- **Firecrawl MCP (SideGuide Technologies, Inc. (d/b/a Firecrawl))**: /search returns titles, descriptions, URLs and highlights at 2 credits per 10 results; scrapeOptions fetches each result as Markdown at 1 credit a page. Keyless MCP exposes search. acc-05 uses it for technical research. No token cap per page.\n- **Olostep (Olostep Technologies Inc.)**: /v1/searches returns up to 25 links with title and description for 5 credits, and scrape_options embeds Markdown per link in the same call with a timeout; a slow link comes back null rather than failing the call.\n- **Outcrawl (Aeonmind LLC (trading as Outcrawl))**: Search returns ranked results at 5 credits per 10, pro rata, and scrape: true renders each hit through the scrape pipeline at 1 credit in the same call; a search every engine refuses bills nothing.\n- **Oxylabs Web Scraper API (oxylabs, UAB)**: POST /v1/search returns ranked results with max_results and geo parameters, then /v1/scrape with output markdown reads each page; the agent guide documents this search-then-scrape pattern.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Diffbot (Diffbot Technologies Corp.)**: Web Search costs 1 credit a search and the product page says a response can carry a page's full content, but the /docs/search page returned 404 so the parameters and output shape weren't read.\n- **Jina Reader (Jina AI GmbH)**: s.jina.ai returns the top five results with each page already read as Markdown, but keyless cannot search, every call costs at least 10,000 tokens, x-token-budget is ignored on search, and acc-31 saw single requests of 1.9M tokens.\n- **Qovia (Qovia)**: POST /v1/search returns ranked items with text from ten Chinese social and news sources for $0.005 a call with paging, but there is no page fetch and it covers only the listed platforms, not the open web.\n- **Scrape.do (PACKEND, LLC)**: Google Search API returns parsed SERP JSON at 10 credits a request, then each page can be fetched with output=markdown. No token or character cap on the page read.\n- **ScrapeGraphAI (ScrapeGraphAI, Inc.)**: search returns up to 20 results with each page as Markdown in prune mode in one call at 2 credits a result. No token or character budget on the returned pages.\n- **Scrapeless (NST LABS TECH LTD.)**: Google Search API returns parsed results by q, start and num; pages then go through the scrape job for markdown. No token cap, and the per-query price was not on the pages read.\n- **ScrapeOps Proxy API Aggregator (ScrapeOps Limited)**: No search endpoint; a Google results page can be fetched and parsed with auto_extract, but the parser's cost isn't in the request-cost table and there is no token cap.\n- **ScraperAPI (ScraperAPI LLC)**: Google SERP structured endpoint returns JSON (25 credits a request) and pages read with output_format=markdown; no token or character cap.\n- **Scrapewise (BEBOTECH OÜ)**: google_search returns web, news or shopping results at 10 credits a query and fetch_page reads each page as text at 5 to 25 credits; no token cap.\n- **ScrapingBee (VostokInc SAS)**: Google Search API returns organic_results and more at 10 credits (15 with light_request=false), then return_page_markdown reads each page. No token cap and multi-page billing is unstated.\n- **Scrapingdog (OpenData Labs Private Limited)**: Google Search API returns parsed results at 5 to 10 credits, then /scrape with markdown=true reads pages; no token cap.\n- **Spider (BAGELMEN LLC)**: /search returns ranked results and, with fetch_page_content, each page as Markdown in one call at bytes-plus-compute pricing. No token cap; search_limit bounds result count only.\n- **WebScraping.AI (Urlooker LLC)**: /serp returns organic results with snippets at 15 credits a page and /text reads each page as Markdown; response_too_large asks you to fall back to /selected and there is no token cap.\n- **Zyte API (Zyte Group Limited)**: The Search API returns typed organicResults for a query and domain, and markdown: true reads each page; no token cap.\n\n## Couldn't check\n\nThe docs were closed to us, or silent. These carry no mark either way.\n\n- **Microlink (Microlink Labs, S.L.)**: The MCP code registers a search tool and the listing carries search.serp, but no search page was found in the docs index read and the OpenAPI file has one path.\n\n- not this job: BrowserAct (Hongkong Ziniao Technology Co., Limited): No search endpoint or tool; the MCP exposes bot and template search, task status and cancel only.\n- not this job: jevcrawl (Rayen Mlayeh): No web search; crawl, scrape and map all start from a URL the agent already has.\n- not this job: ScrapeAtlas (OD. Media \u0026 Technology LLC): No web search endpoint; keyword search exists only inside platforms (YouTube, Reddit, Quora) and returns records, not pages.\n- not this job: Scrapfly (Joam Intelligence, LLC): No web search endpoint; the docs_search MCP tool searches Scrapfly's own docs.\n- not this job: ScrapingAnt (DATAANT SP. Z O.O.): No search endpoint; Google pages come back as raw HTML at 10 credits through /v2/general.\n- not this job: SoldFetch (SoldFetch): No web search; searches are eBay listing queries.\n- not this job: ZenRows (ZENROWS, S.L.): No search endpoint among the five primitives.\n\n1 listings in the category were added after this reading and aren't marked yet.\n\n## What stops agents finishing\n\n- search endpoint bills a fixed token minimum per call and spikes on large pages\n- keyless tier cannot search\n- search results return redirect URLs instead of destinations\n\n## Around it\n\nBefore: search.serp. After: llm.synthesise, web-scrapers.page-to-markdown.\n\n## Where this comes from\n\nWeight 10 of 100 in Web scraping \u0026 crawling: 6 of 35 accounts; agents described as combining a search step with page reads (SearXNG plus Crawl4AI, Tavily plus Firecrawl, s.jina.ai)\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/web-scrapers.search-then-read.json, or the MCP tool get_use_case with id web-scrapers.search-then-read. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Research loop: find pages, then read the few that matter",
        "url": ""
      }
    ],
    "description": "Research loop: find pages, then read the few that matter: 10 companies fully do it, 14 with a limit, 77 carry the tag. Every listing in Web scraping \u0026 crawling with the reason and a source.",
    "facts": null,
    "h1": "Research loop: find pages, then read the few that matter",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/web-scrapers.search-then-read/",
    "published": "",
    "section": "agents",
    "title": "Research loop: find pages, then read the few that matter | Anchor…",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/web-scrapers.search-then-read/"
  },
  "tokens": {
    "markdown": 2200,
    "slim": 430
  },
  "version": 1
}
