{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/web-scrapers.docs-site-to-knowledge/",
    "json": "https://www.anchorterminal.com/niches/web-scrapers.docs-site-to-knowledge/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/web-scrapers.docs-site-to-knowledge/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/web-scrapers.docs-site-to-knowledge/index.min.md"
  },
  "markdown": "Four or more companies' docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 7 | 9 | 40 | 12 | 6 |\n\n## What counts as done\n\nOne Markdown file per page with its source URL, boilerplate removed, finished in one job the agent can poll.\n\nMeasured, for our own tests: the page count is within 10 per cent of the sitemap.\n\nHow agents ask for it: pull the whole docs site into markdown; index these docs so I can ask questions; give my coding agent the current docs for this library\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **Apify MCP Server (Apify Technologies s.r.o.)**: Website Content Crawler takes maxCrawlPages, strips navigation and footers, outputs Markdown per page with its URL, and call-actor returns run and dataset ids after 45 seconds for get-actor-run and get-dataset-items polling; acc-23 published a docs-to-RAG Actor the same way.\n- **Context.dev (Context Dev Inc.)**: Crawl takes maxPages (default 100, maximum 500), maxDepth and useMainContentOnly and returns Markdown per page at 1 credit; larger sites go through batches, which take a sitemap or start URL and are polled by id.\n- **fastCRW (Nanofy OÜ)**: POST /v1/crawl returns an id, maxPages defaults to 100 (10 over MCP) with maxDepth 2, GET /v1/crawl/{id} reports scraping, completed or failed with partial data, each page carries Markdown with onlyMainContent true, and map reads sitemap.xml by default.\n- **Decodo Web Data API (UAB \"Data troops\")**: /crawl returns Markdown by default, takes limit (1 to 10,000), max_depth and max_duration, reads sitemaps, exposes a status URL and per-URL task list, keeps results 24 hours and charges one premium request per page; failed pages aren't charged.\n- **Firecrawl MCP (SideGuide Technologies, Inc. (d/b/a Firecrawl))**: POST /v2/crawl reads sitemaps, returns a job id, polls with next paging, keeps results 24 hours and gives Markdown per page with metadata.sourceURL. limit defaults to 10,000 pages, so set it; GET /crawl/{id}/errors lists pages Firecrawl itself failed on.\n- **Olostep (Olostep Technologies Inc.)**: /v1/crawls requires max_pages, takes max_depth and include or exclude globs, is polled by id until completed, lists pages with a cursor and returns each page's Markdown through /v1/retrieve at 1 credit a page.\n- **ScrapeGraphAI (ScrapeGraphAI, Inc.)**: crawl is async with a job id, maxPages default 50 (max 1,000), maxDepth, include and exclude patterns, markdown per page and a cursor-paged pages endpoint. 2 credits to start plus 1 a page.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Bright Data (Bright Data Ltd.)**: No link-following crawl among the 30 MCP tools; scrape_batch takes five URLs and the Fetch API's async trigger takes URL lists. The pricing page lists a Crawl API from $1 per 1,000 whose docs weren't read.\n- **Crawl4AI (CONTEXT4AI PTE LTD)**: Crawl4AI Cloud has no link-following crawl; /scrape/jobs takes a URL list you must build. The open-source library crawls, and acc-10 built a docs indexer on it, but you host it yourself.\n- **Diffbot (Diffbot Technologies Corp.)**: Crawl spiders a site for 0 credits plus 1 per extracted page and Bulk takes URL lists, but Crawl needs the Plus plan ($899 a month) and output is typed JSON text, not Markdown with links.\n- **Outcrawl (Aeonmind LLC (trading as Outcrawl))**: Crawl streams Markdown per page from a seed, answers a job id at once and is polled with status, cursor results and cancel, but dropping the HTTP stream cancels the crawl, limit has no stated default, and sitemap reading isn't described.\n- **Scrapeless (NST LABS TECH LTD.)**: Crawl takes limit (default 10,000), maxDepth, include and exclude paths and sitemap handling, outputs markdown with onlyMainContent, and is polled by job id. The 10,000 default and GB-plus-hour billing make cost hard to bound.\n- **ScraperAPI (ScraperAPI LLC)**: Crawler API runs a polled job bounded by max_depth or crawl_budget in credits with regexp filters, results kept 7 days. Free plan allows depth 1, and no Markdown or main-content option is documented for crawl output.\n- **Scrapfly (Joam Intelligence, LLC)**: Crawler API takes a seed URL with sitemap discovery and page_limit, returns a UUID to poll and WARC or HAR artifacts; a Markdown-per-page output and main-content option were not confirmed on the page read.\n- **Spider (BAGELMEN LLC)**: /crawl with return_format markdown, readability, depth and a sitemap-aware engine returns one record per page with its URL, but limit defaults to your credit balance and results stream back rather than through a polled job.\n- **ZenRows (ZENROWS, S.L.)**: Crawl takes max_pages (default 100, up to 100,000), output_format markdown, cursor-paged results, an Idempotency-Key and a polled status, but the Markdown keeps boilerplate and a challenge page is crawled like any other.\n\n- not this job: BrowserAct (Hongkong Ziniao Technology Co., Limited): No crawler or sitemap job; workflow bots follow pagination and loop nodes a person designs, one site at a time.\n- not this job: jevcrawl (Rayen Mlayeh): crawl follows a question and returns only the relevant passages (default max_pages 30, return_tokens 4,000); map returns ranked URLs. It doesn't aim at exporting every page of a site.\n- not this job: Jina Reader (Jina AI GmbH): No crawl, map or job endpoint; Reader takes one URL per request. acc-15 and acc-16 convert documentation page by page and feed it to a model themselves.\n- not this job: Microlink (Microlink Labs, S.L.): No crawl, map or job endpoint; the API is one URL per GET.\n- not this job: Oxylabs Web Scraper API (oxylabs, UAB): The Web API has scrape, search and dedicated scrapers but no link-following crawl or sitemap job in the pages read.\n- not this job: Qovia (Qovia): No crawl or page endpoint.\n- not this job: Scrape.do (PACKEND, LLC): No crawl or link-following endpoint. The Async API takes a list of target URLs you already have, and there is no main-content filter.\n- not this job: ScrapeAtlas (OD. Media \u0026 Technology LLC): No crawl endpoint and no Markdown output.\n- not this job: ScrapeOps Proxy API Aggregator (ScrapeOps Limited): No crawl or link-following endpoint in the Proxy API Aggregator.\n- not this job: Scrapewise (BEBOTECH OÜ): AI scrapers turn a link list or sitemap into rows of fields, not one Markdown file per page.\n- not this job: ScrapingAnt (DATAANT SP. Z O.O.): No crawl endpoint; one request is one page.\n- not this job: ScrapingBee (VostokInc SAS): No crawl endpoint; one call is one URL.\n- not this job: Scrapingdog (OpenData Labs Private Limited): No crawl endpoint in the 60-plus APIs.\n- not this job: SoldFetch (SoldFetch): No crawl and no Markdown.\n- not this job: WebScraping.AI (Urlooker LLC): No crawl endpoint; one call is one URL.\n- not this job: Zyte API (Zyte Group Limited): Zyte API has no crawl endpoint; link following belongs to Scrapy Cloud.\n\n1 listings in the category were added after this reading and aren't marked yet.\n\n## What stops agents finishing\n\n- crawl has no default page limit and runs to the credit balance\n- map stops early so pages are missed\n- job cannot be polled from the MCP within the tool-call timeout\n\n## Around it\n\nBefore: search.find-url. After: vector-databases.embed, document-extraction.chunk.\n\n## Where this comes from\n\nWeight 12 of 100 in Web scraping \u0026 crawling: 6 of 35 accounts; the recurring agent-builder job on Hacker News (Doctor, HyperResearch, llm-min, Apify docs Actor)\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/web-scrapers.docs-site-to-knowledge.json, or the MCP tool get_use_case with id web-scrapers.docs-site-to-knowledge. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Crawl a documentation or help site into Markdown for a knowledge base",
        "url": ""
      }
    ],
    "description": "Crawl a documentation or help site into Markdown for a knowledge base: 7 companies fully do it, 9 with a limit, 40 carry the tag. Every listing in Web scraping \u0026 crawling with the reason and a source.",
    "facts": null,
    "h1": "Crawl a documentation or help site into Markdown for a knowledge base",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/web-scrapers.docs-site-to-knowledge/",
    "published": "",
    "section": "agents",
    "title": "Crawl a documentation or help site into Markdown for a knowledge base…",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/web-scrapers.docs-site-to-knowledge/"
  },
  "tokens": {
    "markdown": 2150,
    "slim": 330
  },
  "version": 1
}
