AI Crawler Index · category
Do reference websites block AI crawlers?
As of 2026-11-16, 27.9% of the 43 reference websites whose robots.txt we could read block GPTBot, 23.3% block ClaudeBot and 9.3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1 of 53 publish an llms.txt file.
Block rate by crawler: reference vs all sites
| Crawler | Operator | Reference | All sites |
|---|---|---|---|
| GPTBot | OpenAI | 27.9% | 15.4% |
| CCBot | Common Crawl | 27.9% | 16.4% |
| ClaudeBot | Anthropic | 23.3% | 14.7% |
| Applebot-Extended | Apple | 20.9% | 13.4% |
| Bytespider | ByteDance | 20.9% | 16.2% |
| Meta-ExternalAgent | Meta | 18.6% | 13.6% |
| Google-Extended | 16.3% | 13.2% | |
| Amazonbot | Amazon | 14% | 13.2% |
| cohere-ai | Cohere | 14% | 12.7% |
| ChatGPT-User | OpenAI | 11.6% | 7.4% |
| PerplexityBot | Perplexity | 9.3% | 8.2% |
| Google-CloudVertexBot | 9.3% | 8.8% | |
| OAI-SearchBot | OpenAI | 7% | 5.8% |
| Claude-SearchBot | Anthropic | 7% | 5.7% |
| Claude-User | Anthropic | 7% | 5.9% |
| Perplexity-User | Perplexity | 7% | 6.1% |
| meta-externalfetcher | Meta | 7% | 6.5% |
| MistralAI-Training | Mistral | 7% | 7.1% |
| OAI-AdsBot | OpenAI | 4.7% | 3% |
| Google-GeminiNotebook | 4.7% | 3.3% | |
| Google-Agent | 4.7% | 3.6% | |
| meta-webindexer | Meta | 4.7% | 5.1% |
| meta-externalads | Meta | 4.7% | 2.9% |
| Amzn-SearchBot | Amazon | 4.7% | 4.4% |
| Amzn-User | Amazon | 4.7% | 4.4% |
| MistralAI-User | Mistral | 4.7% | 5.7% |
| MistralAI-Index | Mistral | 4.7% | 3.6% |
| DuckAssistBot | DuckDuckGo | 2.3% | 5.9% |
| Applebot | Apple | 0% | 3.2% |
All 53 reference sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 16 | wikipedia.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 64 | archive.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 79 | w3.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 92 | doi.org | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 96 | sciencedirect.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 99 | researchgate.net | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 109 | wikimedia.org | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 114 | who.int | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 115 | imdb.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 122 | springer.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 126 | nature.com | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 136 | wiley.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 203 | arxiv.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 205 | goodreads.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 212 | ietf.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 215 | quora.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | partial | allowed | – |
| 222 | fandom.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 227 | stackoverflow.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 238 | oup.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 248 | openstreetmap.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 249 | tandfonline.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 270 | ieee.org | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 274 | mdpi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 287 | sagepub.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 294 | britannica.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 301 | cambridge.org | Blocks some AI search | allowed | allowed | blocked | allowed | allowed | allowed | allowed | allowed | – |
| 310 | iso.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 323 | merriam-webster.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 378 | academia.edu | Blocks some AI search | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 390 | frontiersin.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 396 | biblegateway.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 405 | mayoclinic.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 417 | genius.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 432 | webmd.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 440 | rfc-editor.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 454 | healthline.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 471 | letterboxd.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 526 | apa.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 532 | ultimate-guitar.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 538 | dafont.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 539 | plos.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 544 | jw.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 549 | namu.wiki | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 573 | wiktionary.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 604 | themoviedb.org | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 607 | psychologytoday.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 608 | mydramalist.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 620 | bmj.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 632 | jstor.org | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 634 | calculator.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 647 | biomedcentral.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 675 | stackexchange.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 688 | elsevier.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
FAQ
- Do reference websites block AI crawlers?
- As of 2026-11-16, 27.9% of the 43 reference websites whose robots.txt we could read block GPTBot, 23.3% block ClaudeBot and 9.3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1 of 53 publish an llms.txt file.
- Which reference websites block GPTBot?
- imdb.com, nature.com, goodreads.com, quora.com, tandfonline.com, academia.edu, webmd.com, healthline.com, letterboxd.com, themoviedb.org, and more (full list on this page).
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%