{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/web-scrapers.bulk-page-collection/",
    "json": "https://www.anchorterminal.com/niches/web-scrapers.bulk-page-collection/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/web-scrapers.bulk-page-collection/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/web-scrapers.bulk-page-collection/index.min.md"
  },
  "markdown": "Four or more companies' docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 8 | 9 | 41 | 8 | 4 |\n\n## What counts as done\n\nA batch job id returned at once, results pageable with a cursor, and a cost estimate given before the job.\n\nMeasured, for our own tests: at least 95 per cent of URLs returned, and the total cost within 10 per cent of the estimate.\n\nHow agents ask for it: scrape all 300,000 listings; fetch every product page in this category; batch these 5,000 URLs\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **Apify MCP Server (Apify Technologies s.r.o.)**: call-actor returns a run id at once, get-dataset-items pages with limit and offset, maxItems caps paid results, and acc-25 reports hundreds of runs a month; ACTOR_MAX_PAID_DATASET_ITEMS stops a runaway run. Dataset retention wasn't read.\n- **Context.dev (Context Dev Inc.)**: Batches take up to 100,000 URLs with an Idempotency-Key, return an id at once, poll by status, page results with next_cursor, keep files 180 days, reserve credits at submit and refund unused pages.\n- **Crawl4AI (CONTEXT4AI PTE LTD)**: POST /scrape/jobs takes up to 10,000 URLs, returns a job id at once, pages results with ?after=N (500 a page), refunds URLs that got no answer, retries failures, and POST /v1/estimate prices the job first.\n- **Diffbot (Diffbot Technologies Corp.)**: Bulk Extract takes a batch of URLs, runs asynchronously, is managed by job (pause, resume, delete) and downloaded as a collection; cost is 1 credit a page and inactive jobs are deleted ten days after completion.\n- **Olostep (Olostep Technologies Inc.)**: /v1/batches takes up to 100,000 URLs, returns an id and status at once, pages items with a cursor, completes even when some URLs fail, keeps results about 7 days unless storage.expires_in extends them, and costs 1 credit a URL. New accounts are capped at 100 items a batch.\n- **Scrapewise (BEBOTECH OÜ)**: scrapewise_run_scraper_url_list takes up to 5,000 addresses as one run returning a job_id for get_results, with quote_only for cost and failed pages free. Data kept 90 days.\n- **Scrapfly (Joam Intelligence, LLC)**: Crawler url_list mode runs a list as one polled job with page_limit and artifacts; /scrape/batch streams up to 100 configs per call on paid plans with atomic concurrency checks. cost=true estimates before a scrape.\n- **ZenRows (ZENROWS, S.L.)**: Batch takes up to 100,000 URLs per job, returns a job_id at once, pages results by cursor, keeps them 14 days, dedupes retries with Idempotency-Key and estimates credits from the fixed rate card.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Bright Data (Bright Data Ltd.)**: The Fetch API's asynchronous /trigger route is recommended for batch jobs and run_scraper returns snapshot ids, but result paging, retention and a cost estimate weren't in the pages read; the MCP's scrape_batch stops at five URLs.\n- **Decodo Web Data API (UAB \"Data troops\")**: POST /v1/task queues a request and returns a task_id, results stay 24 hours and prices are fixed per 1,000, but there is no cursor, no estimate and no documented batch concurrency beyond the plan rate limit.\n- **Firecrawl MCP (SideGuide Technologies, Inc. (d/b/a Firecrawl))**: Batch scrape returns a job id, pages with next and skip, reports creditsUsed and takes maxConcurrency, but results stay readable only 24 hours and there is no estimate endpoint; cost is predictable at 1 credit a page as each page completes.\n- **Oxylabs Web Scraper API (oxylabs, UAB)**: POST /v1/async/scrape/batch/{source} takes up to 5,000 inputs for selected sources, returns 202 with job ids and errors[], and results come by polling, callback or cloud storage; no cursor paging, retention period or estimate was in the pages read.\n- **Scrape.do (PACKEND, LLC)**: Async API returns a JobID and TaskIDs at once and is polled per job and task, with its own concurrency pool (2 on Free, 30% of plan). No cost estimate before a job, no cursor over results and no stated retention.\n- **ScrapeGraphAI (ScrapeGraphAI, Inc.)**: No URL-list batch; crawl stops at 1,000 pages per job and follows links rather than taking a list. Pages are cursor-paged and cost is 2 plus 1 a page.\n- **Scrapeless (NST LABS TECH LTD.)**: Batch scrape takes a URL list and returns a job with status and errors endpoints to poll, but no cost estimate, no stated retention and billing by traffic and browser hours.\n- **ScraperAPI (ScraperAPI LLC)**: POST /batchjobs takes up to 50,000 URLs and returns one job per URL with a statusUrl to poll; only 200 and 404 are billed and /account/urlcost estimates a domain. No cursor over results and no stated retention.\n- **Spider (BAGELMEN LLC)**: /crawl takes limit, budget per path and concurrency_limit, with costs per page and Unlimited plans by seat; no job id to poll, results come back in the response or by webhook, and limit is unbounded by default.\n\n- not this job: BrowserAct (Hongkong Ziniao Technology Co., Limited): No URL-list batch; each bot run is one task, with 2 to 40 simultaneous tasks by plan.\n- not this job: fastCRW (Nanofy OÜ): No URL-list page batch: /v1/scrape takes one URL, /v1/crawl follows links from a seed, and /v1/extract takes a URL list only for LLM extraction at 5 credits a page.\n- not this job: jevcrawl (Rayen Mlayeh): No batch job; crawl is bounded by max_pages and max_seconds and returns passages, not a dataset.\n- not this job: Jina Reader (Jina AI GmbH): No batch or async job endpoint for URL lists; each page is one request at 20 requests a minute keyless or 500 with a key.\n- not this job: Microlink (Microlink Labs, S.L.): No batch endpoint; one URL per request at 2 concurrent on Free and plan quotas that pause at 100 per cent.\n- not this job: Outcrawl (Aeonmind LLC (trading as Outcrawl)): No URL-list batch: scrape takes one URL and crawl walks a site from a seed. Crawl results page with a cursor and limit bounds spend, but the input is a seed, not a list.\n- not this job: Qovia (Qovia): No batch job; each call returns one page of records and datasets are quoted by hand.\n- not this job: ScrapeAtlas (OD. Media \u0026 Technology LLC): No job or URL-list endpoint; batch operations exist only for Bluesky profiles and posts. Four concurrent requests per account.\n- not this job: ScrapeOps Proxy API Aggregator (ScrapeOps Limited): No batch or async job endpoint; one request per URL, with 1 concurrent thread on the Free plan.\n- not this job: ScrapingAnt (DATAANT SP. Z O.O.): No batch or job endpoint; Free accounts get 409 on a second concurrent request.\n- not this job: ScrapingBee (VostokInc SAS): No batch or async job endpoint; each URL is a synchronous call within the plan's concurrency.\n- not this job: Scrapingdog (OpenData Labs Private Limited): No batch endpoint; a 202 means one request was queued, and the Free plan allows one concurrent request.\n- not this job: SoldFetch (SoldFetch): Jobs are capped at 100 pages with one active job per workspace and 24-hour retention, so thousands of pages are out of scope.\n- not this job: WebScraping.AI (Urlooker LLC): No batch or async job endpoint; 2 concurrent requests on the Free plan.\n- not this job: Zyte API (Zyte Group Limited): No batch or job endpoint in Zyte API; the agent loops requests under the 3,000 RPM limit, as acc-19 did for 300,000 listings with their own stack.\n\n1 listings in the category were added after this reading and aren't marked yet.\n\n## What stops agents finishing\n\n- free tier allows one concurrent request\n- results expire after about a week\n- no cost estimate before a large job\n\n## Around it\n\nBefore: web-scrapers.docs-site-to-knowledge, search.serp. After: databases.insert, document-extraction.table-from-text.\n\n## Where this comes from\n\nWeight 8 of 100 in Web scraping \u0026 crawling: 4 of 35 accounts; Japan real estate, fashion market intel, Apify volumes and the Firecrawl niche scrape\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/web-scrapers.bulk-page-collection.json, or the MCP tool get_use_case with id web-scrapers.bulk-page-collection. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Collect thousands of pages into a dataset",
        "url": ""
      }
    ],
    "description": "Collect thousands of pages into a dataset: 8 companies fully do it, 9 with a limit, 41 carry the tag. Every listing in Web scraping \u0026 crawling with the reason and a source.",
    "facts": null,
    "h1": "Collect thousands of pages into a dataset",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/web-scrapers.bulk-page-collection/",
    "published": "",
    "section": "agents",
    "title": "Collect thousands of pages into a dataset | Anchor Terminal",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/web-scrapers.bulk-page-collection/"
  },
  "tokens": {
    "markdown": 2250,
    "slim": 380
  },
  "version": 1
}
