TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do other popular websites block AI crawlers?

As of 2026-11-16, 15.1% of the 6,651 other popular websites whose robots.txt we could read block GPTBot, 14.1% block ClaudeBot and 7.8% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1185 of 7963 publish an llms.txt file.

Block rate by crawler: other popular vs all sites

CrawlerOperatorOtherAll sites
CCBotCommon Crawl16%16.4%
BytespiderByteDance15.6%16.2%
GPTBotOpenAI15.1%15.4%
ClaudeBotAnthropic14.1%14.7%
Meta-ExternalAgentMeta13.3%13.6%
Google-ExtendedGoogle13%13.2%
Applebot-ExtendedApple12.9%13.4%
AmazonbotAmazon12.9%13.2%
cohere-aiCohere12.3%12.7%
Google-CloudVertexBotGoogle8.9%8.8%
PerplexityBotPerplexity7.8%8.2%
ChatGPT-UserOpenAI7.2%7.4%
MistralAI-TrainingMistral7.1%7.1%
meta-externalfetcherMeta6.2%6.5%
Perplexity-UserPerplexity5.7%6.1%
DuckAssistBotDuckDuckGo5.7%5.9%
OAI-SearchBotOpenAI5.6%5.8%
Claude-UserAnthropic5.6%5.9%
MistralAI-UserMistral5.6%5.7%
Claude-SearchBotAnthropic5.3%5.7%
meta-webindexerMeta5%5.1%
Amzn-UserAmazon4.5%4.4%
Amzn-SearchBotAmazon4.4%4.4%
Google-AgentGoogle3.6%3.6%
MistralAI-IndexMistral3.6%3.6%
Google-GeminiNotebookGoogle3.4%3.3%
ApplebotApple3.3%3.2%
OAI-AdsBotOpenAI3%3%
meta-externaladsMeta3%2.9%

Top 150 of 7,963 other popular sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
10dzen.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
14azure.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
18cloud.microsoftNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
20gandi.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
45dellemc.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
47windows.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
54reg.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
75base.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
76macromedia.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
77hosting24.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
80flickr.comBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
82creativecommons.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
113roku.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
117launchpad.netBlocks some AI searchblockedblockedallowedblockedblockedblockedblockedblocked–
119mobile.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
123googleblog.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
138drom.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
139crpt.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
142aliyun.comOpenpartialallowedallowedpartialallowedallowedallowedpartial–
143name.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
151trustpilot.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialpartial–
155selectel.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
167netangels.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
175indeed.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
179timeweb.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
182go.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedallowed–
206amazonalexa.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
213cloudflareinsights.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
219bitdefender.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
220gosuslugi.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
231eventbrite.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
233share.googleNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
236bitrix24.ruOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
237capcut.comOpenpartialpartialpartialpartialpartialpartialpartialpartialyes
241blogger.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
256oraclecloud.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
271ovh.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
298yelp.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
304vkontakte.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
305porkbun.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
308wpguardian.ioOpenallowedallowedallowedallowedallowedallowedallowedallowed–
309consultant.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
311uk.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
314outbrain.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
320fb.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
321oxylabs.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
331webempresa.euNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
337beget.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
367discordapp.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
373usps.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
376pccc.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
385pendo.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
387life360.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
399it.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
406agora.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
418deloitte.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
421smartthings.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
457eu.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
472kontur.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
504safety.googleOpenallowedallowedallowedallowedallowedallowedallowedallowed–
514lge.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
521superhosting.bgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
535inven.co.krOpenallowedallowedallowedallowedallowedallowedallowedallowed–
536one.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
541locaweb.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
546inleed.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
565tabor.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
572hostpoint.chOpenallowedallowedallowedallowedallowedallowedallowedallowed–
588us.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5910xrpc.ioNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
593jusbrasil.com.brBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialblockedyes
598beboo.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
600ap4r.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
602litnet.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
609naukri.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
610bbb.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
611id.meUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
623grow.googleOpenallowedallowedallowedallowedallowedallowedallowedallowed–
627labs.googleOpenallowedallowedallowedallowedallowedallowedallowedallowed–
628edna.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
635jino.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
661bookmyshow.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
665mbga.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
686sotwe.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
690home.plOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
695cmoa.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
696platform.shOpenallowedallowedallowedallowedallowedallowedallowedallowed–
701max.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
705nosv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
723testbook.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7271mg.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
729immobiliare.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
733notion.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowedyes
734manulife-sinochem.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
735absher.saNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
736bt.dkBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
737xhwide.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
738programme-tv.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblockedyes
740aajtak.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
742iau.irUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
743hover.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
744goodreturns.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
745kakuyomu.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
747ppomppu.co.krOpenallowedallowedallowedallowedallowedallowedallowedallowed–
748deezer.comOpenpartialpartialpartialpartialpartialpartialpartialpartial–
749tjk.orgBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
751rakuten-sec.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
754bikewale.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
756trilltrill.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
757aka.msBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
761therapservices.netOpenpartialpartialpartialpartialallowedpartialpartialpartialyes
762sinoptik.uaOpenallowedallowedallowedallowedallowedallowedallowedallowed–
763buzzfeed.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
764wordwall.netOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
765autoplius.ltUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
766theqoo.netBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
767acm.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
768mongodb.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
769screener.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
770mamastar.jpBlocks training onlyallowedallowedallowedpartialallowedallowedallowedallowed–
772chordtela.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
773tupi.fmOpenallowedallowedallowedallowedallowedallowedallowedallowed–
774mechacomic.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
775girlschannel.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
776czbooks.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
777freewebnovel.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
778ss.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
780site.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
783skelbiu.ltUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
785rayjump.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
787eepurl.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
788smadex.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
789piccoma.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
790alphapolis.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
791lecturas.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
792rambler.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
793wpastra.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
794letras.mus.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
795publi24.roUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
796afribaba.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
797animanch.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
801twkan.comBlocks some AI searchblockedallowedblockedblockedallowedallowedblockedblocked–
803eff.orgOpenpartialallowedallowedpartialallowedallowedpartialpartialyes
804dto.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
805arbada.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
808goal7.coOpenallowedallowedallowedallowedallowedallowedallowedallowed–
809ecnavi.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
812promiedos.com.arNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
813tjsp.jus.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
815turbo.azUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.

FAQ

Do other popular websites block AI crawlers?
As of 2026-11-16, 15.1% of the 6,651 other popular websites whose robots.txt we could read block GPTBot, 14.1% block ClaudeBot and 7.8% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1185 of 7963 publish an llms.txt file.
Which other popular websites block GPTBot?
flickr.com, launchpad.net, go.com, fb.com, litnet.com, bt.dk, tjk.org, aka.ms, buzzfeed.com, theqoo.net, and more (full list on this page).

Other categories