Niches · Web scraping & crawling
Collect thousands of pages into a dataset
Four or more companies' docs show a full fit.
What counts as done
A batch job id returned at once, results pageable with a cursor, and a cost estimate given before the job.
Measured, for our own tests: at least 95 per cent of URLs returned, and the total cost within 10 per cent of the estimate. No docs we read state these figures, so they don't count against a listing here.
How agents ask for it: scrape all 300,000 listings
, fetch every product page in this category
, batch these 5,000 URLs
Does the job 8
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| Apify MCP Server A Apify Technologies s.r.o. | call-actor returns a run id at once, get-dataset-items pages with limit and offset, maxItems caps paid results, and acc-25 reports hundreds of runs a month; ACTOR_MAX_PAID_DATASET_ITEMS stops a runaway run. Dataset retention wasn't read. source |
| Context.dev A Context Dev Inc. | Batches take up to 100,000 URLs with an Idempotency-Key, return an id at once, poll by status, page results with next_cursor, keep files 180 days, reserve credits at submit and refund unused pages. source |
| Crawl4AI D CONTEXT4AI PTE LTD | POST /scrape/jobs takes up to 10,000 URLs, returns a job id at once, pages results with ?after=N (500 a page), refunds URLs that got no answer, retries failures, and POST /v1/estimate prices the job first. source |
| Diffbot C Diffbot Technologies Corp. | Bulk Extract takes a batch of URLs, runs asynchronously, is managed by job (pause, resume, delete) and downloaded as a collection; cost is 1 credit a page and inactive jobs are deleted ten days after completion. source |
| Olostep B Olostep Technologies Inc. | /v1/batches takes up to 100,000 URLs, returns an id and status at once, pages items with a cursor, completes even when some URLs fail, keeps results about 7 days unless storage.expires_in extends them, and costs 1 credit a URL. New accounts are capped at 100 items a batch. source |
| Scrapewise B BEBOTECH OÜ | scrapewise_run_scraper_url_list takes up to 5,000 addresses as one run returning a job_id for get_results, with quote_only for cost and failed pages free. Data kept 90 days. source |
| Scrapfly BB Joam Intelligence, LLC | Crawler url_list mode runs a list as one polled job with page_limit and artifacts; /scrape/batch streams up to 100 configs per call on paid plans with atomic concurrency checks. cost=true estimates before a scrape. source |
| ZenRows A ZENROWS, S.L. | Batch takes up to 100,000 URLs per job, returns a job_id at once, pages results by cursor, keeps them 14 days, dedupes retries with Idempotency-Key and estimates credits from the fixed rate card. source |
Does the job, with a limit 9
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Bright Data C Bright Data Ltd. | The Fetch API's asynchronous /trigger route is recommended for batch jobs and run_scraper returns snapshot ids, but result paging, retention and a cost estimate weren't in the pages read; the MCP's scrape_batch stops at five URLs. source |
| Decodo Web Data API B UAB "Data troops" | POST /v1/task queues a request and returns a task_id, results stay 24 hours and prices are fixed per 1,000, but there is no cursor, no estimate and no documented batch concurrency beyond the plan rate limit. source |
| Firecrawl MCP A SideGuide Technologies, Inc. (d/b/a Firecrawl) | Batch scrape returns a job id, pages with next and skip, reports creditsUsed and takes maxConcurrency, but results stay readable only 24 hours and there is no estimate endpoint; cost is predictable at 1 credit a page as each page completes. source |
| Oxylabs Web Scraper API B oxylabs, UAB | POST /v1/async/scrape/batch/{source} takes up to 5,000 inputs for selected sources, returns 202 with job ids and errors[], and results come by polling, callback or cloud storage; no cursor paging, retention period or estimate was in the pages read. source |
| Scrape.do E PACKEND, LLC | Async API returns a JobID and TaskIDs at once and is polled per job and task, with its own concurrency pool (2 on Free, 30% of plan). No cost estimate before a job, no cursor over results and no stated retention. source |
| ScrapeGraphAI B ScrapeGraphAI, Inc. | No URL-list batch; crawl stops at 1,000 pages per job and follows links rather than taking a list. Pages are cursor-paged and cost is 2 plus 1 a page. source |
| Scrapeless C NST LABS TECH LTD. | Batch scrape takes a URL list and returns a job with status and errors endpoints to poll, but no cost estimate, no stated retention and billing by traffic and browser hours. source |
| ScraperAPI D ScraperAPI LLC | POST /batchjobs takes up to 50,000 URLs and returns one job per URL with a statusUrl to poll; only 200 and 404 are billed and /account/urlcost estimates a domain. No cursor over results and no stated retention. source |
| Spider BB BAGELMEN LLC | /crawl takes limit, budget per path and concurrency_limit, with costs per page and Unlimited plans by seat; no job id to poll, results come back in the response or by webhook, and limit is unbounded by default. source |
Doesn't do this job 15
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| BrowserAct D Hongkong Ziniao Technology Co., Limited | No URL-list batch; each bot run is one task, with 2 to 40 simultaneous tasks by plan. source |
| fastCRW B Nanofy OÜ | No URL-list page batch: /v1/scrape takes one URL, /v1/crawl follows links from a seed, and /v1/extract takes a URL list only for LLM extraction at 5 credits a page. source |
| jevcrawl B Rayen Mlayeh | No batch job; crawl is bounded by max_pages and max_seconds and returns passages, not a dataset. source |
| Jina Reader C Jina AI GmbH | No batch or async job endpoint for URL lists; each page is one request at 20 requests a minute keyless or 500 with a key. source |
| Microlink B Microlink Labs, S.L. | No batch endpoint; one URL per request at 2 concurrent on Free and plan quotas that pause at 100 per cent. source |
| Outcrawl B Aeonmind LLC (trading as Outcrawl) | No URL-list batch: scrape takes one URL and crawl walks a site from a seed. Crawl results page with a cursor and limit bounds spend, but the input is a seed, not a list. source |
| Qovia D Qovia | No batch job; each call returns one page of records and datasets are quoted by hand. source |
| ScrapeAtlas B OD. Media & Technology LLC | No job or URL-list endpoint; batch operations exist only for Bluesky profiles and posts. Four concurrent requests per account. source |
| ScrapeOps Proxy API Aggregator D ScrapeOps Limited | No batch or async job endpoint; one request per URL, with 1 concurrent thread on the Free plan. source |
| ScrapingAnt E DATAANT SP. Z O.O. | No batch or job endpoint; Free accounts get 409 on a second concurrent request. source |
| ScrapingBee B VostokInc SAS | No batch or async job endpoint; each URL is a synchronous call within the plan's concurrency. source |
| Scrapingdog E OpenData Labs Private Limited | No batch endpoint; a 202 means one request was queued, and the Free plan allows one concurrent request. source |
| SoldFetch E SoldFetch | Jobs are capped at 100 pages with one active job per workspace and 24-hour retention, so thousands of pages are out of scope. source |
| WebScraping.AI B Urlooker LLC | No batch or async job endpoint; 2 concurrent requests on the Free plan. source |
| Zyte API BB Zyte Group Limited | No batch or job endpoint in Zyte API; the agent loops requests under the 3,000 RPM limit, as acc-19 did for 300,000 listings with their own stack. source |
1 listings in the category were added after this reading and aren't marked yet.
What stops agents finishing
- free tier allows one concurrent request
- results expire after about a week
- no cost estimate before a large job
Around it
Before: web-scrapers.docs-site-to-knowledge, search.serp. After: databases.insert, document-extraction.table-from-text.
Where this comes from
Weight 8 of 100 in Web scraping & crawling: 4 of 35 accounts; Japan real estate, fashion market intel, Apify volumes and the Firecrawl niche scrape
The same as data: /api/v1/use-cases/web-scrapers.bulk-page-collection.json, or the MCP tool get_use_case with id web-scrapers.bulk-page-collection. Other jobs in this category: /api/v1/niches?category=web-scrapers.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.