TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do tech websites block AI crawlers?

As of 2026-10-01, 4.5% of the 154 tech websites whose robots.txt we could read block GPTBot, 5.2% block ClaudeBot and 2.6% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 55 of 173 publish an llms.txt file.

Block rate by crawler: tech vs all sites

CrawlerOperatorTechAll sites
BytespiderByteDance7.1%17.7%
CCBotCommon Crawl5.8%19.5%
ClaudeBotAnthropic5.2%16.3%
GPTBotOpenAI4.5%17%
Google-ExtendedGoogle3.9%14%
PerplexityBotPerplexity2.6%12.7%
Applebot-ExtendedApple2.6%14.3%
cohere-aiCohere2.6%13.7%
Meta-ExternalAgentMeta1.9%14.4%
ChatGPT-UserOpenAI1.3%9.6%
OAI-AdsBotOpenAI1.3%4.3%
Claude-SearchBotAnthropic1.3%8.1%
Google-GeminiNotebookGoogle1.3%4.3%
meta-webindexerMeta1.3%6.3%
meta-externaladsMeta1.3%3.8%
Amzn-SearchBotAmazon1.3%5.9%
Amzn-UserAmazon1.3%5.5%
MistralAI-IndexMistral1.3%4.7%
MistralAI-TrainingMistral1.3%4.7%
OAI-SearchBotOpenAI0.6%6.7%
Claude-UserAnthropic0.6%8.3%
Perplexity-UserPerplexity0.6%8.9%
Google-CloudVertexBotGoogle0.6%7.2%
Google-AgentGoogle0.6%4.5%
meta-externalfetcherMeta0.6%8.7%
AmazonbotAmazon0.6%13.1%
DuckAssistBotDuckDuckGo0.6%8.1%
MistralAI-UserMistral0.6%8.4%
ApplebotApple0%2.6%

All 173 tech sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
1google.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2cloudflare.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4microsoft.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6apple.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8mail.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
11office.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
14github.comBlocks training onlypartialpartialallowedpartialallowedpartialallowedallowedyes
16bing.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
20wordpress.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
23icloud.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
27yahoo.comBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
30adobe.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
31qq.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
32zoom.usOpenallowedallowedallowedallowedallowedallowedallowedallowed–
33nginx.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
34baidu.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
38opera.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
39f5.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
40sentry.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
42samsung.comOpenpartialpartialpartialallowedallowedpartialpartialallowedyes
43mozilla.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
47ui.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
48office365.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
53apache.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
55dropbox.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
58unity3d.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
61kaspersky.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
63godaddy.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
66xiaomi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
68webex.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknownyes
88oracle.comOpenpartialpartialpartialpartialallowedpartialpartialallowed–
90android.comOpenpartialpartialallowedpartialpartialpartialpartialpartial–
93ubuntu.comOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
99cisco.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
102stripe.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
104canva.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
105cpanel.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
106sourceforge.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
107meraki.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedpartial–
108gmail.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
111hubspot.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
115namecheap.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
117ibm.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
121hp.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
122autodesk.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
128debian.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
131avast.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
133dell.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
137php.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
140hostinger.comBlocks training onlypartialpartialpartialpartialpartialpartialpartialblockedyes
142mi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
145gnu.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
146weebly.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
147arubanetworks.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
149naver.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
150slack.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
152checkpoint.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
157gitlab.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
158hostgator.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
159zendesk.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
161nvidia.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
162tp-link.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
163issuu.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
168plesk.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
169calendly.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblockedyes
174unsplash.comBlocks some AI searchallowedallowedallowedblockedblockedallowedallowedallowed–
175163.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
176brave.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
182netease.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
187huawei.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
190paloaltonetworks.comOpenallowedpartialpartialallowedpartialpartialallowedallowedyes
191duckduckgo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
196duolingo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
197aol.comBlocks training onlyblockedallowedpartialblockedallowedpartialblockedblocked–
198pixabay.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
201bluehost.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
202speedtest.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
204mikrotik.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
205sina.com.cnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
206sohu.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialpartial–
209okta.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
213addtoany.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
216mediatek.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
218docker.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
219scribd.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
227mysql.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
230slideshare.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
235trendmicro.comOpenallowedpartialpartialpartialpartialpartialpartialpartialyes
236squarespace.comOpenpartialallowedallowedpartialallowedallowedpartialpartialyes
239firefox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
241fast.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
242digitalocean.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
243grammarly.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
249eset.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
254free.frOpenallowedallowedallowedallowedallowedallowedallowedallowed–
255teamviewer.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
256dreamhost.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
261klaviyo.comOpenpartialpartialpartialpartialpartialpartialallowedallowedyes
270atlassian.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
274wix.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
278tencent.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
279mega.co.nzOpenallowedallowedallowedallowedallowedallowedallowedallowed–
280jotform.comOpenpartialallowedallowedpartialallowedpartialallowedpartial–
283linode.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
286intercom.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
289redhat.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
293360.cnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
295surveymonkey.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
296anydesk.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
301sophos.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
314wps.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
315ring.comOpenpartialallowedpartialpartialallowedpartialpartialallowedyes
316seznam.czOpenallowedallowedallowedallowedallowedallowedallowedallowed–
321ya.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
325asus.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
328mailchimp.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
330python.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
338mediafire.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
340box.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
346lenovo.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
361vivo.com.cnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
362zoho.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
364odoo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
368me.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
370blackberry.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
377daum.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
379pexels.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
381jetbrains.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
384figma.comBlocks training onlyblockedpartialpartialblockedpartialpartialblockedblocked–
387synology.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
388cloudinary.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
394disqus.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
398lg.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
405arcgis.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
408starlink.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
411qualtrics.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
421tailscale.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
434xerox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
445statcounter.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
449jimdo.comOpenpartialpartialpartialpartialpartialpartialpartialpartialyes
453ecosia.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
455nextcloud.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
457spamhaus.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
465typeform.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
466freepik.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
467garmin.comBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedallowed–
475supabase.coOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
477adblockplus.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
479netgear.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
491ilovepdf.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
493instructure.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
505magnific.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
514service-now.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
519meethue.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
527auth0.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
529tesla.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
549classlink.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
553siemens.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
556remove.bgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
557coursera.orgBlocks training onlypartialpartialpartialpartialallowedpartialpartialpartialyes
569savefrom.netOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
573kakao.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
580ovhcloud.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
606postgresql.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
611clever.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
640fdown.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
647kernel.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
659twilio.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
664rustdesk.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblockedyes
696ssstik.ioOpenallowedallowedallowedallowedallowedallowedallowedallowed–
704utorrent.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
715fedoraproject.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
887udemy.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

FAQ

Do tech websites block AI crawlers?
As of 2026-10-01, 4.5% of the 154 tech websites whose robots.txt we could read block GPTBot, 5.2% block ClaudeBot and 2.6% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 55 of 173 publish an llms.txt file.
Which tech websites block GPTBot?
yahoo.com, canva.com, meraki.com, aol.com, pixabay.com, pexels.com, figma.com.

Other categories