TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · domain suffix

Do .com websites block AI crawlers?

As of 2026-10-02, 12.4% of the 4,470 .com websites whose robots.txt we could read block GPTBot, 12.2% block ClaudeBot and 9% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 868 of them (19.4%) serve an llms.txt (all sites: 15.2%).

12.4%
block GPTBot (all sites 12%)
12.2%
block ClaudeBot (all sites 11%)
19.4%
serve an llms.txt (all sites 15.2%)
4,470
of 5,298 sites with a readable robots.txt

Block rate by crawler: .com vs all sites

CrawlerOperator.comAll sitesDifference
BytespiderByteDance14%12.2%+1.8 pp
CCBotCommon Crawl14%12.7%+1.3 pp
GPTBotOpenAI12.4%12%+0.4 pp
ClaudeBotAnthropic12.2%11%+1.2 pp
Meta-ExternalAgentMeta10.8%9.5%+1.3 pp
Applebot-ExtendedApple10.7%9.6%+1.1 pp
Google-ExtendedGoogle10.2%9.5%+0.7 pp
AmazonbotAmazon10%9.2%+0.8 pp
cohere-aiCohere10%9%+1 pp
PerplexityBotPerplexity9%8.3%+0.7 pp
ChatGPT-UserOpenAI7.8%7.4%+0.4 pp
meta-externalfetcherMeta6.8%6%+0.8 pp
DuckAssistBotDuckDuckGo6.6%5.7%+0.9 pp
Perplexity-UserPerplexity6.6%5.7%+0.9 pp
Claude-UserAnthropic6.4%5.5%+0.9 pp
Claude-SearchBotAnthropic6.2%5.4%+0.8 pp
MistralAI-UserMistral6.1%5.3%+0.8 pp
Google-CloudVertexBotGoogle6%5.2%+0.8 pp
OAI-SearchBotOpenAI5.8%5.7%+0.1 pp
meta-webindexerMeta5.6%4.6%+1 pp
Amzn-SearchBotAmazon5.2%4.1%+1.1 pp
Amzn-UserAmazon4.8%3.9%+0.9 pp
MistralAI-IndexMistral3.8%3.3%+0.5 pp
Google-AgentGoogle3.6%3.2%+0.4 pp
MistralAI-TrainingMistral3.5%3.2%+0.3 pp
Google-GeminiNotebookGoogle3.3%3%+0.3 pp
OAI-AdsBotOpenAI3.3%3%+0.3 pp
meta-externaladsMeta3.1%2.8%+0.3 pp
ApplebotApple3%3.1%-0.1 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open3,07358%
Unreadable82815.6%
No robots.txt56310.6%
Blocks some AI search4819.1%
Blocks training only2855.4%
Blocks all AI search681.3%

Most restrictive .com sites

  1. reddit.com 29 of 29 blocked
  2. sportfilm800.com 29 of 29 blocked
  3. qcloud.com 29 of 29 blocked
  4. adrta.com 29 of 29 blocked
  5. constellix.com 29 of 29 blocked
  6. co.com 29 of 29 blocked
  7. juicyscore.com 29 of 29 blocked
  8. tappx.com 29 of 29 blocked
  9. urldefense.com 29 of 29 blocked
  10. system-monitor.com 29 of 29 blocked
  11. jwpltx.com 29 of 29 blocked
  12. hellotalk8.com 29 of 29 blocked
  13. lenovomm.com 29 of 29 blocked
  14. sportradarserving.com 29 of 29 blocked
  15. backblazeb2.com 29 of 29 blocked

Sites with an llms.txt (868)

Top 100 of 5,298 .com sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
1google.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2cloudflare.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3facebook.comBlocks some AI searchpartialblockedblockedpartialblockedpartialpartialblocked–
4microsoft.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5youtube.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6apple.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7instagram.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9twitter.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
11linkedin.comBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
12office.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
13amazon.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
14azure.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
15github.comBlocks training onlypartialpartialallowedpartialallowedpartialallowedallowedyes
17bing.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
19netflix.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialblocked–
21x.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
23skype.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
24icloud.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
25pinterest.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
26tiktok.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
27whatsapp.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedallowed–
28yahoo.comBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
29roblox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
30adobe.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
31spotify.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
33qq.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
35baidu.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
36chatgpt.comBlocks some AI searchallowedallowedallowedallowedallowedblockedblockedblocked–
37opera.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
38vimeo.comBlocks training onlypartialallowedallowedpartialallowedallowedpartialblocked–
39openai.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
40f5.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
45dellemc.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
46samsung.comOpenpartialpartialpartialallowedallowedpartialpartialallowedyes
47windows.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
48reddit.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
49ui.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
50office365.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
52amazonvideo.comBlocks some AI searchallowedallowedblockedallowedallowedblockedallowedblocked–
53epicgames.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
55snapchat.comBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
57dropbox.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
58intuit.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
59vk.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
62unity3d.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
63kaspersky.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
65godaddy.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
66tumblr.comBlocks training onlyallowedallowedallowedblockedallowedallowedblockedblocked–
67nytimes.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
68xiaomi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
70webex.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknownyes
71paypal.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
72shopify.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
73spectrum.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
76macromedia.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
77hosting24.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
78paytmfirstgames.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
80flickr.comBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
81soundcloud.comOpenpartialpartialpartialpartialallowedpartialpartialpartial–
83medium.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedallowed–
84cnn.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
85theguardian.comBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
86ebay.comBlocks some AI searchblockedpartialpartialblockedpartialblockedallowedblocked–
87android.comOpenpartialpartialallowedpartialpartialpartialpartialpartial–
91oracle.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
93ubuntu.comOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
94forbes.comBlocks some AI searchblockedallowedallowedblockedallowedblockedallowedblocked–
95bbc.comBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
96sciencedirect.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
100cisco.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
104stripe.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
106canva.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
110meraki.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedpartial–
111gmail.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
112hubspot.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
113roku.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
115imdb.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
116namecheap.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
118booking.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
120ibm.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
121reuters.comBlocks some AI searchblockedpartialpartialblockedblockedblockedblockedblocked–
122springer.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
123googleblog.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
124hp.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
125autodesk.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
126nature.comBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
128steampowered.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
129weibo.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
130avast.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
134alibaba.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedallowed–
135weather.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedblockedyes
136wiley.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
137dell.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
141etsy.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
142aliyun.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
143name.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
144mi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
145weebly.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
146bilibili.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
148arubanetworks.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .com

FAQ

Do .com websites block AI crawlers?
As of 2026-10-02, 12.4% of the 4,470 .com websites whose robots.txt we could read block GPTBot, 12.2% block ClaudeBot and 9% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 868 of them (19.4%) serve an llms.txt (all sites: 15.2%).
Which .com websites block the most AI crawlers?
reddit.com (29 of 29), sportfilm800.com (29 of 29), qcloud.com (29 of 29), adrta.com (29 of 29), constellix.com (29 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 5,298 .com sites are in the sample, 4,470 had a readable robots.txt (828 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes