TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do reference websites block AI crawlers?

As of 2026-10-01, 27.9% of the 43 reference websites whose robots.txt we could read block GPTBot, 23.3% block ClaudeBot and 9.3% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 1 of 53 publish an llms.txt file.

Block rate by crawler: reference vs all sites

CrawlerOperatorReferenceAll sites
GPTBotOpenAI27.9%17%
CCBotCommon Crawl27.9%19.5%
ClaudeBotAnthropic23.3%16.3%
BytespiderByteDance20.9%17.7%
Applebot-ExtendedApple18.6%14.3%
Meta-ExternalAgentMeta18.6%14.4%
Google-ExtendedGoogle16.3%14%
AmazonbotAmazon14%13.1%
cohere-aiCohere14%13.7%
ChatGPT-UserOpenAI11.6%9.6%
PerplexityBotPerplexity9.3%12.7%
Google-CloudVertexBotGoogle9.3%7.2%
OAI-SearchBotOpenAI7%6.7%
Claude-SearchBotAnthropic7%8.1%
Claude-UserAnthropic7%8.3%
Perplexity-UserPerplexity7%8.9%
meta-externalfetcherMeta7%8.7%
MistralAI-TrainingMistral7%4.7%
OAI-AdsBotOpenAI4.7%4.3%
Google-GeminiNotebookGoogle4.7%4.3%
Google-AgentGoogle4.7%4.5%
meta-webindexerMeta4.7%6.3%
meta-externaladsMeta4.7%3.8%
Amzn-SearchBotAmazon4.7%5.9%
Amzn-UserAmazon4.7%5.5%
MistralAI-UserMistral4.7%8.4%
MistralAI-IndexMistral4.7%4.7%
ApplebotApple2.3%2.6%
DuckAssistBotDuckDuckGo2.3%8.1%

All 53 reference sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
15wikipedia.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
62archive.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
77w3.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
89doi.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
94sciencedirect.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
96researchgate.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
109wikimedia.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
112who.intBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
113imdb.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
120springer.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
123nature.comBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
136wiley.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
200goodreads.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
203arxiv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
211ietf.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
212quora.comBlocks some AI searchblockedblockedblockedblockedblockedblockedpartialallowed–
221stackoverflow.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
223fandom.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
233oup.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
245tandfonline.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
246openstreetmap.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
266ieee.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
268mdpi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
285sagepub.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
290britannica.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
299cambridge.orgBlocks some AI searchallowedallowedblockedallowedallowedallowedallowedallowed–
307iso.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
332merriam-webster.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
374academia.eduBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
386frontiersin.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
392biblegateway.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
401mayoclinic.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
413genius.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
427webmd.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
442healthline.comBlocks some AI searchblockedallowedallowedblockedallowedallowedallowedblocked–
451rfc-editor.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
468letterboxd.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
517apa.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
521ultimate-guitar.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
528dafont.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
531plos.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
535jw.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
539namu.wikiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
566wiktionary.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
593themoviedb.orgBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
596psychologytoday.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
601mydramalist.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
617bmj.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
623jstor.orgBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
625calculator.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
631biomedcentral.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
682elsevier.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
694stackexchange.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

FAQ

Do reference websites block AI crawlers?
As of 2026-10-01, 27.9% of the 43 reference websites whose robots.txt we could read block GPTBot, 23.3% block ClaudeBot and 9.3% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 1 of 53 publish an llms.txt file.
Which reference websites block GPTBot?
imdb.com, nature.com, goodreads.com, quora.com, tandfonline.com, academia.edu, webmd.com, healthline.com, letterboxd.com, themoviedb.org, and more (full list on this page).

Other categories