Niches · Web scraping & crawling

Collect thousands of pages into a dataset

Four or more companies' docs show a full fit.

8companies, full fit
9companies, fit with a limit
41companies carry the tag
8share of the category's work
4first-hand accounts

What counts as done

A batch job id returned at once, results pageable with a cursor, and a cost estimate given before the job.

Measured, for our own tests: at least 95 per cent of URLs returned, and the total cost within 10 per cent of the estimate. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: scrape all 300,000 listings, fetch every product page in this category, batch these 5,000 URLs

Does the job 8

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
Apify MCP Server A
Apify Technologies s.r.o.
call-actor returns a run id at once, get-dataset-items pages with limit and offset, maxItems caps paid results, and acc-25 reports hundreds of runs a month; ACTOR_MAX_PAID_DATASET_ITEMS stops a runaway run. Dataset retention wasn't read. source
Context.dev A
Context Dev Inc.
Batches take up to 100,000 URLs with an Idempotency-Key, return an id at once, poll by status, page results with next_cursor, keep files 180 days, reserve credits at submit and refund unused pages. source
Crawl4AI D
CONTEXT4AI PTE LTD
POST /scrape/jobs takes up to 10,000 URLs, returns a job id at once, pages results with ?after=N (500 a page), refunds URLs that got no answer, retries failures, and POST /v1/estimate prices the job first. source
Diffbot C
Diffbot Technologies Corp.
Bulk Extract takes a batch of URLs, runs asynchronously, is managed by job (pause, resume, delete) and downloaded as a collection; cost is 1 credit a page and inactive jobs are deleted ten days after completion. source
Olostep B
Olostep Technologies Inc.
/v1/batches takes up to 100,000 URLs, returns an id and status at once, pages items with a cursor, completes even when some URLs fail, keeps results about 7 days unless storage.expires_in extends them, and costs 1 credit a URL. New accounts are capped at 100 items a batch. source
Scrapewise B
BEBOTECH OÜ
scrapewise_run_scraper_url_list takes up to 5,000 addresses as one run returning a job_id for get_results, with quote_only for cost and failed pages free. Data kept 90 days. source
Scrapfly BB
Joam Intelligence, LLC
Crawler url_list mode runs a list as one polled job with page_limit and artifacts; /scrape/batch streams up to 100 configs per call on paid plans with atomic concurrency checks. cost=true estimates before a scrape. source
ZenRows A
ZENROWS, S.L.
Batch takes up to 100,000 URLs per job, returns a job_id at once, pages results by cursor, keeps them 14 days, dedupes retries with Idempotency-Key and estimates credits from the fixed rate card. source

Does the job, with a limit 9

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Bright Data C
Bright Data Ltd.
The Fetch API's asynchronous /trigger route is recommended for batch jobs and run_scraper returns snapshot ids, but result paging, retention and a cost estimate weren't in the pages read; the MCP's scrape_batch stops at five URLs. source
Decodo Web Data API B
UAB "Data troops"
POST /v1/task queues a request and returns a task_id, results stay 24 hours and prices are fixed per 1,000, but there is no cursor, no estimate and no documented batch concurrency beyond the plan rate limit. source
Firecrawl MCP A
SideGuide Technologies, Inc. (d/b/a Firecrawl)
Batch scrape returns a job id, pages with next and skip, reports creditsUsed and takes maxConcurrency, but results stay readable only 24 hours and there is no estimate endpoint; cost is predictable at 1 credit a page as each page completes. source
Oxylabs Web Scraper API B
oxylabs, UAB
POST /v1/async/scrape/batch/{source} takes up to 5,000 inputs for selected sources, returns 202 with job ids and errors[], and results come by polling, callback or cloud storage; no cursor paging, retention period or estimate was in the pages read. source
Scrape.do E
PACKEND, LLC
Async API returns a JobID and TaskIDs at once and is polled per job and task, with its own concurrency pool (2 on Free, 30% of plan). No cost estimate before a job, no cursor over results and no stated retention. source
ScrapeGraphAI B
ScrapeGraphAI, Inc.
No URL-list batch; crawl stops at 1,000 pages per job and follows links rather than taking a list. Pages are cursor-paged and cost is 2 plus 1 a page. source
Scrapeless C
NST LABS TECH LTD.
Batch scrape takes a URL list and returns a job with status and errors endpoints to poll, but no cost estimate, no stated retention and billing by traffic and browser hours. source
ScraperAPI D
ScraperAPI LLC
POST /batchjobs takes up to 50,000 URLs and returns one job per URL with a statusUrl to poll; only 200 and 404 are billed and /account/urlcost estimates a domain. No cursor over results and no stated retention. source
Spider BB
BAGELMEN LLC
/crawl takes limit, budget per path and concurrency_limit, with costs per page and Unlimited plans by seat; no job id to poll, results come back in the response or by webhook, and limit is unbounded by default. source

Doesn't do this job 15

Listings in the category that do something else, with the reason.

ListingWhy
BrowserAct D
Hongkong Ziniao Technology Co., Limited
No URL-list batch; each bot run is one task, with 2 to 40 simultaneous tasks by plan. source
fastCRW B
Nanofy OÜ
No URL-list page batch: /v1/scrape takes one URL, /v1/crawl follows links from a seed, and /v1/extract takes a URL list only for LLM extraction at 5 credits a page. source
jevcrawl B
Rayen Mlayeh
No batch job; crawl is bounded by max_pages and max_seconds and returns passages, not a dataset. source
Jina Reader C
Jina AI GmbH
No batch or async job endpoint for URL lists; each page is one request at 20 requests a minute keyless or 500 with a key. source
Microlink B
Microlink Labs, S.L.
No batch endpoint; one URL per request at 2 concurrent on Free and plan quotas that pause at 100 per cent. source
Outcrawl B
Aeonmind LLC (trading as Outcrawl)
No URL-list batch: scrape takes one URL and crawl walks a site from a seed. Crawl results page with a cursor and limit bounds spend, but the input is a seed, not a list. source
Qovia D
Qovia
No batch job; each call returns one page of records and datasets are quoted by hand. source
ScrapeAtlas B
OD. Media & Technology LLC
No job or URL-list endpoint; batch operations exist only for Bluesky profiles and posts. Four concurrent requests per account. source
ScrapeOps Proxy API Aggregator D
ScrapeOps Limited
No batch or async job endpoint; one request per URL, with 1 concurrent thread on the Free plan. source
ScrapingAnt E
DATAANT SP. Z O.O.
No batch or job endpoint; Free accounts get 409 on a second concurrent request. source
ScrapingBee B
VostokInc SAS
No batch or async job endpoint; each URL is a synchronous call within the plan's concurrency. source
Scrapingdog E
OpenData Labs Private Limited
No batch endpoint; a 202 means one request was queued, and the Free plan allows one concurrent request. source
SoldFetch E
SoldFetch
Jobs are capped at 100 pages with one active job per workspace and 24-hour retention, so thousands of pages are out of scope. source
WebScraping.AI B
Urlooker LLC
No batch or async job endpoint; 2 concurrent requests on the Free plan. source
Zyte API BB
Zyte Group Limited
No batch or job endpoint in Zyte API; the agent loops requests under the 3,000 RPM limit, as acc-19 did for 300,000 listings with their own stack. source

1 listings in the category were added after this reading and aren't marked yet.

What stops agents finishing

  • free tier allows one concurrent request
  • results expire after about a week
  • no cost estimate before a large job

Around it

Before: web-scrapers.docs-site-to-knowledge, search.serp. After: databases.insert, document-extraction.table-from-text.

Where this comes from

Weight 8 of 100 in Web scraping & crawling: 4 of 35 accounts; Japan real estate, fashion market intel, Apify volumes and the Firecrawl niche scrape

The same as data: /api/v1/use-cases/web-scrapers.bulk-page-collection.json, or the MCP tool get_use_case with id web-scrapers.bulk-page-collection. Other jobs in this category: /api/v1/niches?category=web-scrapers.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.