TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do government websites block AI crawlers?

As of 2026-10-01, 6.1% of the 33 government websites whose robots.txt we could read block GPTBot, 6.1% block ClaudeBot and 3% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 1 of 52 publish an llms.txt file.

Block rate by crawler: government vs all sites

CrawlerOperatorGovernmentAll sites
AmazonbotAmazon9.1%13.1%
GPTBotOpenAI6.1%17%
ClaudeBotAnthropic6.1%16.3%
Applebot-ExtendedApple6.1%14.3%
CCBotCommon Crawl6.1%19.5%
BytespiderByteDance6.1%17.7%
ChatGPT-UserOpenAI3%9.6%
PerplexityBotPerplexity3%12.7%
Google-ExtendedGoogle3%14%
ApplebotApple3%2.6%
Meta-ExternalAgentMeta3%14.4%
MistralAI-TrainingMistral3%4.7%
OAI-SearchBotOpenAI0%6.7%
OAI-AdsBotOpenAI0%4.3%
Claude-SearchBotAnthropic0%8.1%
Claude-UserAnthropic0%8.3%
Perplexity-UserPerplexity0%8.9%
Google-CloudVertexBotGoogle0%7.2%
Google-GeminiNotebookGoogle0%4.3%
Google-AgentGoogle0%4.5%
meta-webindexerMeta0%6.3%
meta-externalfetcherMeta0%8.7%
meta-externaladsMeta0%3.8%
Amzn-SearchBotAmazon0%5.9%
Amzn-UserAmazon0%5.5%
DuckAssistBotDuckDuckGo0%8.1%
MistralAI-UserMistral0%8.4%
MistralAI-IndexMistral0%4.7%
cohere-aiCohere0%13.7%

All 52 government sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
49europa.euOpenallowedallowedallowedallowedallowedallowedallowedallowed–
59nih.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
79miit.gov.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
101nist.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
125www.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
154cdc.govNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
155un.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
160nasa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
180beian.gov.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
228noaa.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
259ca.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
272unesco.orgBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblockedyes
284loc.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
288service.gov.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
333fda.govNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
372usda.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
400worldbank.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
409epa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
416state.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
431caixa.gov.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
446weforum.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
460whitehouse.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
478maricopa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
487oecd.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
558acesso.gov.brNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
578canada.caOpenallowedallowedallowedallowedallowedallowedallowedallowed–
584admin.chOpenallowedallowedallowedallowedallowedallowedallowedallowed–
612rajasthan.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
628gst.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
637irs.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
638turkiye.gov.trOpenallowedallowedallowedallowedallowedallowedallowedallowed–
657incometax.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
672inps.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
703epfindia.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
706nfse.gov.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
711eci.gov.inNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
737parivahan.gov.inNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
741afip.gob.arUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
748osym.gov.trUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
751ftc.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
773jra.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
786kemkes.go.idNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
798digitallocker.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
802dord.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
814cbl.gov.lyOpenallowedallowedallowedallowedallowedallowedallowedallowed–
858va.govBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
875nps.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
911usgs.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
948sec.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
956meb.gov.trBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
964ed.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
989bls.govNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–

FAQ

Do government websites block AI crawlers?
As of 2026-10-01, 6.1% of the 33 government websites whose robots.txt we could read block GPTBot, 6.1% block ClaudeBot and 3% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 1 of 52 publish an llms.txt file.
Which government websites block GPTBot?
unesco.org, meb.gov.tr.

Other categories