TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do other popular websites block AI crawlers?

As of 2026-10-01, 13.5% of the 260 other popular websites whose robots.txt we could read block GPTBot, 10.8% block ClaudeBot and 8.1% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 48 of 305 publish an llms.txt file.

Block rate by crawler: other popular vs all sites

CrawlerOperatorOtherAll sites
GPTBotOpenAI13.5%17%
CCBotCommon Crawl13.5%19.5%
Meta-ExternalAgentMeta12.3%14.4%
BytespiderByteDance11.5%17.7%
ClaudeBotAnthropic10.8%16.3%
cohere-aiCohere10.8%13.7%
Applebot-ExtendedApple10.4%14.3%
AmazonbotAmazon10.4%13.1%
Google-ExtendedGoogle10%14%
PerplexityBotPerplexity8.1%12.7%
DuckAssistBotDuckDuckGo8.1%8.1%
ChatGPT-UserOpenAI7.7%9.6%
meta-externalfetcherMeta7.7%8.7%
MistralAI-UserMistral7.7%8.4%
Perplexity-UserPerplexity6.9%8.9%
Claude-UserAnthropic6.5%8.3%
meta-webindexerMeta6.5%6.3%
Google-CloudVertexBotGoogle6.2%7.2%
Amzn-SearchBotAmazon6.2%5.9%
Claude-SearchBotAnthropic5.8%8.1%
Google-GeminiNotebookGoogle5.8%4.3%
Google-AgentGoogle5.8%4.5%
Amzn-UserAmazon5.8%5.5%
MistralAI-TrainingMistral5.8%4.7%
OAI-SearchBotOpenAI5.4%6.7%
OAI-AdsBotOpenAI5.4%4.3%
meta-externaladsMeta5.4%3.8%
MistralAI-IndexMistral5.4%4.7%
ApplebotApple2.7%2.6%

All 305 other popular sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
13azure.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
17cloud.microsoftNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
19gandi.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
41windows.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
45dellemc.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
54reg.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
73base.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
74hosting24.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
75macromedia.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
78flickr.comBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
82creativecommons.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
110roku.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
116launchpad.netBlocks some AI searchblockedblockedallowedblockedblockedblockedblockedblocked–
118mobile.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
124googleblog.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
134drom.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
135crpt.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
138aliyun.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
143name.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
151trustpilot.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialblocked–
153selectel.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
164netangels.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
172indeed.comOpenpartialpartialpartialpartialpartialpartialpartialpartial–
177timeweb.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
181go.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedallowed–
193amazonalexa.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
199cloudflareinsights.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
210bitdefender.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
217gosuslugi.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
226eventbrite.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
229share.googleNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
231capcut.comOpenpartialpartialpartialpartialpartialpartialpartialpartialyes
232bitrix24.ruOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
238blogger.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
253oraclecloud.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
269ovh.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
294yelp.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
300vkontakte.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
302uk.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
304wpguardian.ioOpenallowedallowedallowedallowedallowedallowedallowedallowed–
308outbrain.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
311consultant.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
317fb.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
318porkbun.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
319oxylabs.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
331webempresa.euNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
349beget.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
363discordapp.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
369usps.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
378pccc.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
382life360.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
393pendo.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
395it.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
399agora.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
414deloitte.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
417smartthings.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
450eu.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
471kontur.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
499safety.googleOpenallowedallowedallowedallowedallowedallowedallowedallowed–
503lge.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
516superhosting.bgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
526inven.co.krOpenallowedallowedallowedallowedallowedallowedallowedallowed–
533one.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
540inleed.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
554locaweb.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
555tabor.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
571us.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
583jusbrasil.com.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
588ap4r.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
590beboo.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
592litnet.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
600bbb.orgOpenpartialpartialpartialpartialpartialpartialpartialpartialyes
602naukri.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
603id.meUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
613grow.googleOpenallowedallowedallowedallowedallowedallowedallowedallowed–
614edna.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
616labs.googleOpenallowedallowedallowedallowedallowedallowedallowedallowed–
651bookmyshow.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
654mbga.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
660hostpoint.chOpenallowedallowedallowedallowedallowedallowedallowedallowed–
667jino.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
669sotwe.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
689cmoa.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
692nosv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
707platform.shOpenallowedallowedallowedallowedallowedallowedallowedallowed–
709testbook.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7191mg.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
720xhwide.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
722immobiliare.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
723programme-tv.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblockedyes
724bt.dkBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
726absher.saUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
727notion.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowedyes
728aajtak.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
730manulife-sinochem.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
732iau.irUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
733ppomppu.co.krOpenallowedallowedallowedallowedallowedallowedallowedallowed–
734goodreturns.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
735kakuyomu.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
736deezer.comOpenpartialpartialpartialpartialpartialpartialpartialpartial–
738rakuten-sec.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
739tjk.orgBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
740buzzfeed.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
742bikewale.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
743mongodb.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
744trilltrill.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
747aka.msBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
749therapservices.netOpenpartialpartialpartialpartialallowedpartialpartialpartialyes
752autoplius.ltUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
753theqoo.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
754smadex.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
755screener.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
756rayjump.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
757mamastar.jpBlocks training onlyallowedallowedallowedpartialallowedallowedallowedallowed–
758acm.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
759tupi.fmOpenallowedallowedallowedallowedallowedallowedallowedallowed–
761chordtela.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
763wpastra.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
764girlschannel.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
765freewebnovel.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
766czbooks.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
768mechacomic.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
769ss.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
772skelbiu.ltUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
774rambler.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
775piccoma.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
776eepurl.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
777alphapolis.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
778lecturas.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
779letras.mus.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
780publi24.roBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
781site.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
782hik-connect.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
783afribaba.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
785animanch.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
790dto.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
791ecnavi.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
793twkan.comBlocks some AI searchblockedallowedblockedblockedallowedallowedblockedblocked–
794arbada.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
796goal7.coOpenallowedallowedallowedallowedallowedallowedallowedallowed–
797eff.orgOpenpartialallowedallowedpartialallowedallowedpartialpartial–
799promiedos.com.arNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
800tjsp.jus.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
801turbo.azUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
804bancodevenezuela.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
805humoruniv.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
806choiceadvantage.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
807sun.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
808site777.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
810az24.vnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
811linkmake.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
812pagibigfundservices.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8130xrpc.ioNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
815staseraintv.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
816sambad.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
817mobcup.fmBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
819rsch.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
821shutterstock.comBlocks training onlyblockedpartialallowedblockedpartialpartialallowedblocked–
822ad-contents.jpBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
824ico.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
827rocketmoney.devBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
828gartner.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
829akismet.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
830enable-javascript.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
831hover.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
832overwolf.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
833waze.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
835avira.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
836qcloud.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
837ubnt.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
839fao.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
840tawk.toNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
841kargo.comOpenpartialallowedallowedpartialallowedallowedpartialpartial–
842amocrm.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
843contextweb.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
844vmware.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
846w3schools.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
847beeline.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
848medallia.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
849blizzard.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
850acs.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
851adroll.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
852rapid7.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
853wordwall.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
854newyorker.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
855fc2.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
856fedex.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
857trello.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
859register.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
860kraken.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8621c.ruBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
863moodle.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
865blackhub.teamBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
866pewresearch.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
867trustarc.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
869loopia.seOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
871vinted.frBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
873verify.hnOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
874fifa.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
876gdemoideti.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
877sweb.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
878pusher.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
879yimg.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
880sitescout.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
881adrta.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
883sky.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
886elegantthemes.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
889scientificamerican.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
890indexww.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
891clevelandclinic.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
892cpmstar.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
893xinhuanet.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
894norton.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
895orcid.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
896wayground.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
897ixl.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
898vice.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
899merkur.deBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
901samsungknox.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
9021password.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
903fb.watchOpenallowedallowedallowedallowedallowedallowedallowedallowed–
904astrumu.ioOpenallowedallowedallowedallowedallowedallowedallowedallowed–
905sciencemag.orgBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
906thelancet.comBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
908charter.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
909www.nhs.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
912mapbox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
913sprinthost.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
914wikihow.comBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
915git-scm.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
916oreilly.comBlocks training onlyblockedpartialpartialpartialpartialpartialpartialblockedyes
917appier.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
918deepintent.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
919iubenda.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
920bigcommerce.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
921themeisle.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
922moengage.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
923hh.ruOpenpartialpartialpartialpartialallowedpartialpartialpartial–
924hetzner.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
925autotrader.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
926squareup.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
927mindbox.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
928hola.comBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedblocked–
929ueiwsp.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
932salthasher.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
933wa.linkOpenallowedallowedallowedallowedallowedallowedallowedallowed–
934jamanetwork.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
935fastcompany.comBlocks some AI searchblockedpartialpartialblockedpartialblockedpartialblocked–
936strava.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedallowed–
937pnas.orgBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
938ninjarmm.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
939livechatinc.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
940threema.chOpenallowedallowedallowedallowedallowedallowedallowedallowed–
941mitre.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
942auone.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
943m.meNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
944dantri.com.vnOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
945richaudience.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
946rackspace.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
947techtarget.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
949liquipedia.netBlocks some AI searchblockedallowedblockedblockedallowedallowedallowedallowed–
950lnkd.inBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
951iscorp.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
953ssrn.comBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedallowed–
954wpengine.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
955springserve.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
957romarg.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
958privacymanager.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
959freebsd.orgOpenpartialallowedallowedallowedallowedallowedallowedpartial–
960sogou.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
962logitech.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
963connatix.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
965vlr.ggOpenallowedallowedallowedallowedallowedallowedallowedallowed–
966umeng.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
967russianfood.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
968intentiq.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
969truste.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
970acrobat.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
971typepad.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
972science.orgBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
973dondominio.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
974granbluefantasy.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
975wechat.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
976business.siteNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
977getbootstrap.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
978about.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
980sakura.ne.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
981ign.comBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
982wunderground.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
983iponweb.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
984acronis.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
985suruga-ya.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
986matterport.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
987scryfall.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
988eltiempo.esOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
990earthlink.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
991gizmodo.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
992salesforceliveagent.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
993automattic.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
994npmjs.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
995clickpetroleoegas.com.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
996timeanddate.comBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedallowed–
997broadcom.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
998alura.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1000sedoparking.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–

FAQ

Do other popular websites block AI crawlers?
As of 2026-10-01, 13.5% of the 260 other popular websites whose robots.txt we could read block GPTBot, 10.8% block ClaudeBot and 8.1% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 48 of 305 publish an llms.txt file.
Which other popular websites block GPTBot?
flickr.com, launchpad.net, go.com, fb.com, bt.dk, tjk.org, buzzfeed.com, aka.ms, twkan.com, mobcup.fm, and more (full list on this page).

Other categories