TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do reference websites block AI crawlers?

As of 2026-11-16, 27.9% of the 43 reference websites whose robots.txt we could read block GPTBot, 23.3% block ClaudeBot and 9.3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1 of 53 publish an llms.txt file.

Block rate by crawler: reference vs all sites

CrawlerOperatorReferenceAll sites
GPTBotOpenAI27.9%15.4%
CCBotCommon Crawl27.9%16.4%
ClaudeBotAnthropic23.3%14.7%
Applebot-ExtendedApple20.9%13.4%
BytespiderByteDance20.9%16.2%
Meta-ExternalAgentMeta18.6%13.6%
Google-ExtendedGoogle16.3%13.2%
AmazonbotAmazon14%13.2%
cohere-aiCohere14%12.7%
ChatGPT-UserOpenAI11.6%7.4%
PerplexityBotPerplexity9.3%8.2%
Google-CloudVertexBotGoogle9.3%8.8%
OAI-SearchBotOpenAI7%5.8%
Claude-SearchBotAnthropic7%5.7%
Claude-UserAnthropic7%5.9%
Perplexity-UserPerplexity7%6.1%
meta-externalfetcherMeta7%6.5%
MistralAI-TrainingMistral7%7.1%
OAI-AdsBotOpenAI4.7%3%
Google-GeminiNotebookGoogle4.7%3.3%
Google-AgentGoogle4.7%3.6%
meta-webindexerMeta4.7%5.1%
meta-externaladsMeta4.7%2.9%
Amzn-SearchBotAmazon4.7%4.4%
Amzn-UserAmazon4.7%4.4%
MistralAI-UserMistral4.7%5.7%
MistralAI-IndexMistral4.7%3.6%
DuckAssistBotDuckDuckGo2.3%5.9%
ApplebotApple0%3.2%

All 53 reference sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
16wikipedia.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
64archive.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
79w3.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
92doi.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
96sciencedirect.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
99researchgate.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
109wikimedia.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
114who.intBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
115imdb.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
122springer.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
126nature.comBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
136wiley.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
203arxiv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
205goodreads.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
212ietf.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
215quora.comBlocks some AI searchblockedblockedblockedblockedblockedblockedpartialallowed–
222fandom.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
227stackoverflow.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
238oup.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
248openstreetmap.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
249tandfonline.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
270ieee.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
274mdpi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
287sagepub.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
294britannica.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
301cambridge.orgBlocks some AI searchallowedallowedblockedallowedallowedallowedallowedallowed–
310iso.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
323merriam-webster.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
378academia.eduBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
390frontiersin.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
396biblegateway.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
405mayoclinic.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
417genius.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
432webmd.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
440rfc-editor.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
454healthline.comBlocks some AI searchblockedallowedallowedblockedallowedallowedallowedblocked–
471letterboxd.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
526apa.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
532ultimate-guitar.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
538dafont.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
539plos.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
544jw.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
549namu.wikiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
573wiktionary.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
604themoviedb.orgBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
607psychologytoday.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
608mydramalist.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
620bmj.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
632jstor.orgBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
634calculator.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
647biomedcentral.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
675stackexchange.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
688elsevier.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–

FAQ

Do reference websites block AI crawlers?
As of 2026-11-16, 27.9% of the 43 reference websites whose robots.txt we could read block GPTBot, 23.3% block ClaudeBot and 9.3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1 of 53 publish an llms.txt file.
Which reference websites block GPTBot?
imdb.com, nature.com, goodreads.com, quora.com, tandfonline.com, academia.edu, webmd.com, healthline.com, letterboxd.com, themoviedb.org, and more (full list on this page).

Other categories