TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index

29 AI crawlers and how many websites block them

Share of 8,313 popular websites (readable robots.txt) that block each AI crawler on the home page, as of 2026-11-16.

CrawlerOperatorPurposeBlockedTop 100
CCBotCommon CrawlTraining16.4%27.7%
BytespiderByteDanceTraining16.2%30.1%
GPTBotOpenAITraining15.4%22.9%
ClaudeBotAnthropicTraining14.7%25.3%
Meta-ExternalAgentMetaTraining13.6%25.3%
Applebot-ExtendedAppleTraining13.4%25.3%
Google-ExtendedGoogleTraining13.2%20.5%
AmazonbotAmazonTraining13.2%20.5%
cohere-aiCohereTraining12.7%19.3%
Google-CloudVertexBotGoogleAI search8.8%16.9%
PerplexityBotPerplexityAI search8.2%22.9%
ChatGPT-UserOpenAIUser-triggered fetch7.4%15.7%
MistralAI-TrainingMistralTraining7.1%10.8%
meta-externalfetcherMetaUser-triggered fetch6.5%15.7%
Perplexity-UserPerplexityUser-triggered fetch6.1%16.9%
Claude-UserAnthropicUser-triggered fetch5.9%14.5%
DuckAssistBotDuckDuckGoAI search5.9%16.9%
OAI-SearchBotOpenAIAI search5.8%12%
Claude-SearchBotAnthropicAI search5.7%13.3%
MistralAI-UserMistralUser-triggered fetch5.7%14.5%
meta-webindexerMetaAI search5.1%14.5%
Amzn-SearchBotAmazonAI search4.4%13.3%
Amzn-UserAmazonUser-triggered fetch4.4%12%
Google-AgentGoogleUser-triggered fetch3.6%10.8%
MistralAI-IndexMistralAI search3.6%10.8%
Google-GeminiNotebookGoogleUser-triggered fetch3.3%10.8%
ApplebotAppleAI search3.2%1.2%
OAI-AdsBotOpenAIAds review3%12%
meta-externaladsMetaAds review2.9%9.6%