TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do tech websites block AI crawlers?

As of 2026-11-16, 3.7% of the 162 tech websites whose robots.txt we could read block GPTBot, 5.6% block ClaudeBot and 1.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 62 of 183 publish an llms.txt file.

Block rate by crawler: tech vs all sites

CrawlerOperatorTechAll sites
BytespiderByteDance8%16.2%
CCBotCommon Crawl6.2%16.4%
ClaudeBotAnthropic5.6%14.7%
GPTBotOpenAI3.7%15.4%
Google-ExtendedGoogle2.5%13.2%
Applebot-ExtendedApple1.9%13.4%
cohere-aiCohere1.9%12.7%
PerplexityBotPerplexity1.2%8.2%
Meta-ExternalAgentMeta1.2%13.6%
ChatGPT-UserOpenAI0.6%7.4%
OAI-AdsBotOpenAI0.6%3%
Claude-SearchBotAnthropic0.6%5.7%
Google-GeminiNotebookGoogle0.6%3.3%
meta-webindexerMeta0.6%5.1%
meta-externaladsMeta0.6%2.9%
AmazonbotAmazon0.6%13.2%
Amzn-SearchBotAmazon0.6%4.4%
Amzn-UserAmazon0.6%4.4%
MistralAI-IndexMistral0.6%3.6%
MistralAI-TrainingMistral0.6%7.1%
OAI-SearchBotOpenAI0%5.8%
Claude-UserAnthropic0%5.9%
Perplexity-UserPerplexity0%6.1%
Google-CloudVertexBotGoogle0%8.8%
Google-AgentGoogle0%3.6%
ApplebotApple0%3.2%
meta-externalfetcherMeta0%6.5%
DuckAssistBotDuckDuckGo0%5.9%
MistralAI-UserMistral0%5.7%

Top 150 of 183 tech sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
1google.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2cloudflare.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4microsoft.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6apple.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8mail.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
12office.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
15github.comBlocks training onlypartialpartialpartialpartialpartialpartialallowedallowedyes
17bing.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
22wordpress.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
24icloud.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
28yahoo.comBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
30adobe.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
32zoom.usOpenallowedallowedallowedallowedallowedallowedallowedallowed–
33qq.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
34nginx.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
35baidu.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
37opera.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
40f5.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
41sentry.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
42mozilla.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
43yandex.ruOpenallowedpartialallowedallowedallowedallowedallowedallowed–
46samsung.comOpenpartialpartialpartialallowedallowedpartialpartialallowedyes
49ui.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
50office365.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
56apache.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
57dropbox.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
62unity3d.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
63kaspersky.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
65godaddy.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
68xiaomi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
70webex.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknownyes
87android.comOpenpartialpartialallowedpartialpartialpartialpartialpartial–
91oracle.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
93ubuntu.comOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
100cisco.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
104stripe.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
106canva.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
107sourceforge.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
108cpanel.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
110meraki.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
111gmail.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
112hubspot.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
116namecheap.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
120ibm.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
124hp.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
125autodesk.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
130avast.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
132debian.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
137dell.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
140php.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
144mi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
145weebly.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
147gnu.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
148arubanetworks.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
149slack.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
150naver.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
153checkpoint.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
157hostgator.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
161gitlab.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
163zendesk.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
164nvidia.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
165issuu.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
166tp-link.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
171plesk.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
172calendly.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblockedyes
176unsplash.comBlocks some AI searchallowedallowedallowedblockedblockedallowedallowedallowed–
178163.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
181brave.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
184netease.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
190huawei.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
192intel.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
193duckduckgo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
195paloaltonetworks.comOpenallowedpartialpartialallowedpartialpartialallowedallowedyes
197bluehost.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknownyes
198duolingo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
200aol.comBlocks training onlyblockedallowedpartialblockedallowedpartialblockedblocked–
201pixabay.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
204speedtest.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
207mikrotik.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
208okta.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
211sina.com.cnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
214sohu.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialpartial–
216addtoany.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
221mediatek.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
223docker.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
224scribd.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedblocked–
232mysql.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
234slideshare.netBlocks training onlyallowedallowedallowedblockedallowedallowedallowedblocked–
239trendmicro.comOpenallowedpartialpartialpartialpartialpartialpartialpartialyes
240squarespace.comOpenpartialallowedallowedpartialallowedallowedpartialpartialyes
243firefox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
245fast.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
246digitalocean.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
247grammarly.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
254eset.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
255free.frOpenallowedallowedallowedallowedallowedallowedallowedallowed–
258dreamhost.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
260teamviewer.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
266klaviyo.comOpenpartialpartialpartialpartialpartialpartialallowedallowedyes
272atlassian.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
278wix.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
281tencent.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
283linode.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
285jotform.comOpenpartialallowedallowedpartialallowedpartialallowedpartial–
286mega.co.nzOpenallowedallowedallowedallowedallowedallowedallowedallowed–
290intercom.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
293redhat.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
297360.cnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
299surveymonkey.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
302anydesk.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
303sophos.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
318wps.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
319ring.comOpenpartialallowedpartialpartialallowedpartialpartialallowedyes
322seznam.czOpenallowedallowedallowedallowedallowedallowedallowedallowed–
324ya.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
330asus.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
333mailchimp.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
335python.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
341mediafire.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
344box.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
350lenovo.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
364zoho.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
366vivo.com.cnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
370odoo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
372me.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
375blackberry.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
382pexels.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
384daum.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
386jetbrains.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
388figma.comBlocks training onlyblockedpartialpartialblockedpartialpartialblockedblocked–
391cloudinary.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
392synology.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
400disqus.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
403lg.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
409arcgis.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
412starlink.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
414qualtrics.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
424tailscale.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
437xerox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
452statcounter.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
455jimdo.comOpenpartialpartialpartialpartialpartialpartialpartialpartialyes
460spamhaus.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
461ecosia.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
462nextcloud.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
473typeform.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
474garmin.comBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedallowed–
476freepik.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
481supabase.coOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
482instructure.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
483adblockplus.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–

Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.

FAQ

Do tech websites block AI crawlers?
As of 2026-11-16, 3.7% of the 162 tech websites whose robots.txt we could read block GPTBot, 5.6% block ClaudeBot and 1.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 62 of 183 publish an llms.txt file.
Which tech websites block GPTBot?
yahoo.com, canva.com, aol.com, pixabay.com, pexels.com, figma.com.

Other categories