How to convert a website to Markdown for RAG
Give the site's start URL (a docs site, help center or blog) to the Website to Markdown Crawler on Apify and get clean Markdown for every page, RAG chunks with heading paths and one combined OUTPUT.md, for $1.50 per 1,000 pages ($2.50 per 1,000 when a page needs a real browser); failed, blocked and duplicate pages are free.
Last updated
How to do it in 4 steps
- Enter the site in Start URLs (for example
https://docs.example.com/) and set Max pages. - Optional: set RAG chunk size (characters), for example 2000.
- Click Start. Each page becomes one row with
markdownandchunks;OUTPUT.mdholds the whole site in one file. - Load the chunks into your vector store (the README has LangChain and LlamaIndex examples) or fetch them through the Apify API. A 1,000-page docs site costs $1.50 in fast mode.
Specifics
| Input | Start URLs to crawl, or a list of URLs with crawling turned off |
|---|---|
| Output | Markdown per page, chunks with headingPath and charCount, one combined OUTPUT.md; linked PDF, Word, Excel and CSV files converted too |
| Price on Apify | $1.50 per 1,000 pages over plain HTTP, $2.50 per 1,000 with a browser; compute included, no start fee |
| Limits | Up to 10,000 pages per run, public pages only, linked documents up to 15 MB |
| Benchmark (5 sites, 25 pages each, 2026-09-30) | 3–34 seconds per site, at least 97% of sampled body paragraphs kept on every site; Apify's Website Content Crawler on default settings took 56–200 seconds and kept 0% on a Next.js docs site |
| RapidAPI plans | free 300 requests + 20 summaries/month; $9.99/month for 10,000 requests |
FAQ
Can I convert a single page to Markdown for free?
Yes. Open https://tools.yukai.uk/md/<url>: no key or sign-up, PDFs included, up to 100,000 characters, about 10 requests per minute per IP. It uses a plain HTTP fetch, so pages that need JavaScript are not rendered.
Does the crawler work on JavaScript websites?
Yes. In Auto mode, pages that need JavaScript, or sites that block simple requests, are rendered in headless Chrome automatically and charged at the browser price.
How is it different from Apify's Website Content Crawler?
It has a fixed price per page with compute included. Website Content Crawler has more options and bills compute time, so a tuned run can cost less on large sites that work over plain HTTP.
Is the output ready for a vector database?
Yes. With a chunk size set, each page has a chunks array split at paragraph boundaries with overlap, each chunk with its heading path and character count.
Related guides
Related
- Free URL to Markdown (one page, no key)
- Website-to-Markdown benchmark: method and per-run numbers
- All answers
Prices and limits from the Website to Markdown Crawler README on Apify; benchmark figures from our published benchmark page. Prices are the base (Free plan) prices on Apify; larger Apify plans pay less. Actor page: apify.com/tidytools/website-markdown-crawler.