TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do government websites block AI crawlers?

As of 2026-11-16, 3% of the 203 government websites whose robots.txt we could read block GPTBot, 3% block ClaudeBot and 3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 8 of 330 publish an llms.txt file.

Block rate by crawler: government vs all sites

CrawlerOperatorGovernmentAll sites
ChatGPT-UserOpenAI3.9%7.4%
Claude-UserAnthropic3.4%5.9%
Perplexity-UserPerplexity3.4%6.1%
meta-externalfetcherMeta3.4%6.5%
MistralAI-UserMistral3.4%5.7%
GPTBotOpenAI3%15.4%
ClaudeBotAnthropic3%14.7%
PerplexityBotPerplexity3%8.2%
Google-GeminiNotebookGoogle3%3.3%
Google-AgentGoogle3%3.6%
AmazonbotAmazon3%13.2%
Amzn-UserAmazon3%4.4%
CCBotCommon Crawl3%16.4%
BytespiderByteDance3%16.2%
Meta-ExternalAgentMeta2.5%13.6%
cohere-aiCohere2.5%12.7%
OAI-SearchBotOpenAI2%5.8%
Claude-SearchBotAnthropic2%5.7%
Google-ExtendedGoogle2%13.2%
Google-CloudVertexBotGoogle2%8.8%
Applebot-ExtendedApple2%13.4%
meta-webindexerMeta2%5.1%
DuckAssistBotDuckDuckGo2%5.9%
OAI-AdsBotOpenAI1.5%3%
ApplebotApple1.5%3.2%
meta-externaladsMeta1.5%2.9%
Amzn-SearchBotAmazon1.5%4.4%
MistralAI-IndexMistral1.5%3.6%
MistralAI-TrainingMistral1.5%7.1%

Top 150 of 330 government sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
51europa.euOpenallowedallowedallowedallowedallowedallowedallowedallowed–
60nih.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
88miit.gov.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
105nist.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
127www.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
154cdc.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
156un.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
162nasa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
183beian.gov.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
230noaa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
263ca.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
275unesco.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
289loc.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
292service.gov.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
338fda.govNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
374usda.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
402worldbank.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
413epa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
419state.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
435caixa.gov.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
453weforum.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
465whitehouse.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
468bund.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
485maricopa.govOpenpartialpartialallowedpartialpartialpartialallowedpartial–
492oecd.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
564acesso.gov.brNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
587canada.caOpenallowedallowedallowedallowedallowedallowedallowedallowed–
590admin.chOpenallowedallowedallowedallowedallowedallowedallowedallowed–
622rajasthan.gov.inOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
637gst.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
645irs.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
651turkiye.gov.trOpenallowedallowedallowedallowedallowedallowedallowedallowed–
669incometax.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
681uidai.gov.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
682inps.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
715epfindia.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
717nfse.gov.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
720eci.gov.inNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
746indianrail.gov.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
750parivahan.gov.inNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
753afip.gob.arUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
755ftc.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
759osym.gov.trUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
784jra.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
800kemkes.go.idNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
811digitallocker.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
814dord.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
829cbl.gov.lyOpenallowedallowedallowedallowedallowedallowedallowedallowed–
878va.govBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
892nps.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
923meb.gov.trUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
928usgs.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
967www.gob.mxOpenallowedallowedallowedallowedallowedallowedallowedallowed–
980sec.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1012ed.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1031bls.govNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1065hhs.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1090house.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1125justice.govBlocks some AI searchallowedallowedblockedallowedallowedallowedallowedallowed–
1197energy.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1216congress.govBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1220ny.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1249senate.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1297ssa.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1359dhs.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1402archives.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1415nsw.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1421medlineplus.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1463faa.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1479treasury.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1488dot.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1492army.milUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1494fcc.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1500texas.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1509wa.govBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1513privacyshield.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1527usa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1607navy.milUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1642uspto.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1661fbi.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1664cancer.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1680nyc.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1685dol.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1776federalregister.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1784usembassy.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1822dataprivacyframework.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1863mhlw.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1904legislation.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1931osha.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1956cisa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1972sba.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1996clinicaltrials.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2105cia.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2117uscourts.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2120federalreserve.govNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2162uscis.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2235tfl.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2240deped.gov.phUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2243eia.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2321cms.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2354administracionelectronica.gob.esOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2369wcd.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2383serpro.gov.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2387bisp.gov.pkUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2435nationalarchives.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2462af.milUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2482jst.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2501defense.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2598nsf.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2612pa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2617mo.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2715jma.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2756nj.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2839govinfo.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2858nhtsa.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2861cbp.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2927mass.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2975health.milOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2977michigan.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2993ons.gov.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3073ecfr.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3129lbl.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3135legifrance.gouv.frUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3137nla.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3228meti.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3254maryland.govOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3312copyright.govNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3315agenziaentrate.gov.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3355samhsa.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3391mlit.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3422chinatax.gov.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3424gao.govBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedallowed–
3432kerala.gov.inNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3458tn.govUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3464argentina.gob.arOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3554oregon.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3620radio-canada.caOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3679interno.gov.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3681supremecourt.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3697hud.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3738gpo.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3745gsa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3802iowa.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3830consumerfinance.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3889login.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3933fws.govOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3965sunat.gob.peUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4089mofcom.gov.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4101sat.gob.mxUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4162mohfw.gov.inUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.

FAQ

Do government websites block AI crawlers?
As of 2026-11-16, 3% of the 203 government websites whose robots.txt we could read block GPTBot, 3% block ClaudeBot and 3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 8 of 330 publish an llms.txt file.
Which government websites block GPTBot?
congress.gov, wa.gov, nv.gov, educationboardresults.gov.bd, ahrq.gov, vermont.gov.

Other categories