Niches · Web scraping & crawling

Crawl a documentation or help site into Markdown for a knowledge base

Four or more companies' docs show a full fit.

7companies, full fit
9companies, fit with a limit
40companies carry the tag
12share of the category's work
6first-hand accounts

What counts as done

One Markdown file per page with its source URL, boilerplate removed, finished in one job the agent can poll.

Measured, for our own tests: the page count is within 10 per cent of the sitemap. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: pull the whole docs site into markdown, index these docs so I can ask questions, give my coding agent the current docs for this library

Does the job 7

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
Apify MCP Server A
Apify Technologies s.r.o.
Website Content Crawler takes maxCrawlPages, strips navigation and footers, outputs Markdown per page with its URL, and call-actor returns run and dataset ids after 45 seconds for get-actor-run and get-dataset-items polling; acc-23 published a docs-to-RAG Actor the same way. source
Context.dev A
Context Dev Inc.
Crawl takes maxPages (default 100, maximum 500), maxDepth and useMainContentOnly and returns Markdown per page at 1 credit; larger sites go through batches, which take a sitemap or start URL and are polled by id. source
fastCRW B
Nanofy OÜ
POST /v1/crawl returns an id, maxPages defaults to 100 (10 over MCP) with maxDepth 2, GET /v1/crawl/{id} reports scraping, completed or failed with partial data, each page carries Markdown with onlyMainContent true, and map reads sitemap.xml by default. source
Decodo Web Data API B
UAB "Data troops"
/crawl returns Markdown by default, takes limit (1 to 10,000), max_depth and max_duration, reads sitemaps, exposes a status URL and per-URL task list, keeps results 24 hours and charges one premium request per page; failed pages aren't charged. source
Firecrawl MCP A
SideGuide Technologies, Inc. (d/b/a Firecrawl)
POST /v2/crawl reads sitemaps, returns a job id, polls with next paging, keeps results 24 hours and gives Markdown per page with metadata.sourceURL. limit defaults to 10,000 pages, so set it; GET /crawl/{id}/errors lists pages Firecrawl itself failed on. source
Olostep B
Olostep Technologies Inc.
/v1/crawls requires max_pages, takes max_depth and include or exclude globs, is polled by id until completed, lists pages with a cursor and returns each page's Markdown through /v1/retrieve at 1 credit a page. source
ScrapeGraphAI B
ScrapeGraphAI, Inc.
crawl is async with a job id, maxPages default 50 (max 1,000), maxDepth, include and exclude patterns, markdown per page and a cursor-paged pages endpoint. 2 credits to start plus 1 a page. source

Does the job, with a limit 9

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Bright Data C
Bright Data Ltd.
No link-following crawl among the 30 MCP tools; scrape_batch takes five URLs and the Fetch API's async trigger takes URL lists. The pricing page lists a Crawl API from $1 per 1,000 whose docs weren't read. source
Crawl4AI D
CONTEXT4AI PTE LTD
Crawl4AI Cloud has no link-following crawl; /scrape/jobs takes a URL list you must build. The open-source library crawls, and acc-10 built a docs indexer on it, but you host it yourself. source
Diffbot C
Diffbot Technologies Corp.
Crawl spiders a site for 0 credits plus 1 per extracted page and Bulk takes URL lists, but Crawl needs the Plus plan ($899 a month) and output is typed JSON text, not Markdown with links. source
Outcrawl B
Aeonmind LLC (trading as Outcrawl)
Crawl streams Markdown per page from a seed, answers a job id at once and is polled with status, cursor results and cancel, but dropping the HTTP stream cancels the crawl, limit has no stated default, and sitemap reading isn't described. source
Scrapeless C
NST LABS TECH LTD.
Crawl takes limit (default 10,000), maxDepth, include and exclude paths and sitemap handling, outputs markdown with onlyMainContent, and is polled by job id. The 10,000 default and GB-plus-hour billing make cost hard to bound. source
ScraperAPI D
ScraperAPI LLC
Crawler API runs a polled job bounded by max_depth or crawl_budget in credits with regexp filters, results kept 7 days. Free plan allows depth 1, and no Markdown or main-content option is documented for crawl output. source
Scrapfly BB
Joam Intelligence, LLC
Crawler API takes a seed URL with sitemap discovery and page_limit, returns a UUID to poll and WARC or HAR artifacts; a Markdown-per-page output and main-content option were not confirmed on the page read. source
Spider BB
BAGELMEN LLC
/crawl with return_format markdown, readability, depth and a sitemap-aware engine returns one record per page with its URL, but limit defaults to your credit balance and results stream back rather than through a polled job. source
ZenRows A
ZENROWS, S.L.
Crawl takes max_pages (default 100, up to 100,000), output_format markdown, cursor-paged results, an Idempotency-Key and a polled status, but the Markdown keeps boilerplate and a challenge page is crawled like any other. source

Doesn't do this job 16

Listings in the category that do something else, with the reason.

ListingWhy
BrowserAct D
Hongkong Ziniao Technology Co., Limited
No crawler or sitemap job; workflow bots follow pagination and loop nodes a person designs, one site at a time. source
jevcrawl B
Rayen Mlayeh
crawl follows a question and returns only the relevant passages (default max_pages 30, return_tokens 4,000); map returns ranked URLs. It doesn't aim at exporting every page of a site. source
Jina Reader C
Jina AI GmbH
No crawl, map or job endpoint; Reader takes one URL per request. acc-15 and acc-16 convert documentation page by page and feed it to a model themselves. source
Microlink B
Microlink Labs, S.L.
No crawl, map or job endpoint; the API is one URL per GET. source
Oxylabs Web Scraper API B
oxylabs, UAB
The Web API has scrape, search and dedicated scrapers but no link-following crawl or sitemap job in the pages read. source
Qovia D
Qovia
No crawl or page endpoint. source
Scrape.do E
PACKEND, LLC
No crawl or link-following endpoint. The Async API takes a list of target URLs you already have, and there is no main-content filter. source
ScrapeAtlas B
OD. Media & Technology LLC
No crawl endpoint and no Markdown output. source
ScrapeOps Proxy API Aggregator D
ScrapeOps Limited
No crawl or link-following endpoint in the Proxy API Aggregator. source
Scrapewise B
BEBOTECH OÜ
AI scrapers turn a link list or sitemap into rows of fields, not one Markdown file per page. source
ScrapingAnt E
DATAANT SP. Z O.O.
No crawl endpoint; one request is one page. source
ScrapingBee B
VostokInc SAS
No crawl endpoint; one call is one URL. source
Scrapingdog E
OpenData Labs Private Limited
No crawl endpoint in the 60-plus APIs. source
SoldFetch E
SoldFetch
No crawl and no Markdown. source
WebScraping.AI B
Urlooker LLC
No crawl endpoint; one call is one URL. source
Zyte API BB
Zyte Group Limited
Zyte API has no crawl endpoint; link following belongs to Scrapy Cloud. source

1 listings in the category were added after this reading and aren't marked yet.

What stops agents finishing

  • crawl has no default page limit and runs to the credit balance
  • map stops early so pages are missed
  • job cannot be polled from the MCP within the tool-call timeout

Around it

Before: search.find-url. After: vector-databases.embed, document-extraction.chunk.

Where this comes from

Weight 12 of 100 in Web scraping & crawling: 6 of 35 accounts; the recurring agent-builder job on Hacker News (Doctor, HyperResearch, llm-min, Apify docs Actor)

The same as data: /api/v1/use-cases/web-scrapers.docs-site-to-knowledge.json, or the MCP tool get_use_case with id web-scrapers.docs-site-to-knowledge. Other jobs in this category: /api/v1/niches?category=web-scrapers.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.