Which .gov websites block the most AI crawlers?

wa.gov (29 of 29), nv.gov (29 of 29), congress.gov (19 of 29), vermont.gov (7 of 29), ahrq.gov (3 of 29). The full per-site status is on each site page.

How is this measured?

We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 145 .gov sites are in the sample, 108 had a readable robots.txt (37 did not answer or returned an error page).

AI Crawler Index · domain suffix

Do .gov websites block AI crawlers?

As of 2026-10-02, 4.6% of the 108 .gov websites whose robots.txt we could read block GPTBot, 4.6% block ClaudeBot and 4.6% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 2 of them (1.9%) serve an llms.txt (all sites: 15.2%).

4.6%

block GPTBot (all sites 12%)

4.6%

block ClaudeBot (all sites 11%)

1.9%

serve an llms.txt (all sites 15.2%)

108

of 145 sites with a readable robots.txt

Block rate by crawler: .gov vs all sites

Crawler	Operator	.gov	All sites	Difference
Bytespider	ByteDance	4.6%	12.2%	-7.6 pp
CCBot	Common Crawl	4.6%	12.7%	-8.1 pp
ClaudeBot	Anthropic	4.6%	11%	-6.4 pp
GPTBot	OpenAI	4.6%	12%	-7.4 pp
PerplexityBot	Perplexity	4.6%	8.3%	-3.7 pp
Amazonbot	Amazon	3.7%	9.2%	-5.5 pp
ChatGPT-User	OpenAI	3.7%	7.4%	-3.7 pp
cohere-ai	Cohere	3.7%	9%	-5.3 pp
Applebot-Extended	Apple	2.8%	9.6%	-6.8 pp
Claude-SearchBot	Anthropic	2.8%	5.4%	-2.6 pp
Claude-User	Anthropic	2.8%	5.5%	-2.7 pp
DuckAssistBot	DuckDuckGo	2.8%	5.7%	-2.9 pp
Google-CloudVertexBot	Google	2.8%	5.2%	-2.4 pp
Google-Extended	Google	2.8%	9.5%	-6.7 pp
Meta-ExternalAgent	Meta	2.8%	9.5%	-6.7 pp
meta-externalfetcher	Meta	2.8%	6%	-3.2 pp
meta-webindexer	Meta	2.8%	4.6%	-1.8 pp
MistralAI-User	Mistral	2.8%	5.3%	-2.5 pp
OAI-SearchBot	OpenAI	2.8%	5.7%	-2.9 pp
Perplexity-User	Perplexity	2.8%	5.7%	-2.9 pp
Amzn-SearchBot	Amazon	1.9%	4.1%	-2.2 pp
Amzn-User	Amazon	1.9%	3.9%	-2 pp
Applebot	Apple	1.9%	3.1%	-1.2 pp
Google-Agent	Google	1.9%	3.2%	-1.3 pp
Google-GeminiNotebook	Google	1.9%	3%	-1.1 pp
meta-externalads	Meta	1.9%	2.8%	-0.9 pp
MistralAI-Index	Mistral	1.9%	3.3%	-1.4 pp
MistralAI-Training	Mistral	1.9%	3.2%	-1.3 pp
OAI-AdsBot	OpenAI	1.9%	3%	-1.1 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

Policy	Sites	Share
Open	88	60.7%
Unreadable	37	25.5%
No robots.txt	12	8.3%
Blocks some AI search	4	2.8%
Blocks training only	2	1.4%
Blocks all AI search	2	1.4%

Sites with an llms.txt (2)

Top 100 of 145 .gov sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#	Site	Policy	GPTBot	OAI-Search	ChatGPT-User	ClaudeBot	Claude-Search	Perplexity	Google-Ext.	CCBot	llms.txt
60	nih.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
105	nist.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
154	cdc.gov	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
162	nasa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
230	noaa.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
263	ca.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
289	loc.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
338	fda.gov	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
374	usda.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
413	epa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
419	state.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
465	whitehouse.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
485	maricopa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
645	irs.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
755	ftc.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
878	va.gov	Blocks training only	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
892	nps.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
928	usgs.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
980	sec.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1012	ed.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1031	bls.gov	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1065	hhs.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1090	house.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1125	justice.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1197	energy.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1216	congress.gov	Blocks some AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
1220	ny.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1249	senate.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1297	ssa.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1359	dhs.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1402	archives.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1421	medlineplus.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1463	faa.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1479	treasury.gov	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1488	dot.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1494	fcc.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1500	texas.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1509	wa.gov	Blocks all AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
1513	privacyshield.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1527	usa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1642	uspto.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1661	fbi.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1664	cancer.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1680	nyc.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1685	dol.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1776	federalregister.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1784	usembassy.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1822	dataprivacyframework.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1931	osha.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1956	cisa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1972	sba.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1996	clinicaltrials.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2105	cia.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2117	uscourts.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2120	federalreserve.gov	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2162	uscis.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2243	eia.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2321	cms.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2501	defense.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2598	nsf.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2612	pa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2617	mo.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2756	nj.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2839	govinfo.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2858	nhtsa.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2861	cbp.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2927	mass.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2977	michigan.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
3073	ecfr.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3129	lbl.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3254	maryland.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
3312	copyright.gov	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3355	samhsa.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
3424	gao.gov	Blocks some AI search	allowed	allowed	allowed	allowed	allowed	blocked	allowed	allowed	–
3458	tn.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
3554	oregon.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3681	supremecourt.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3697	hud.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3738	gpo.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3745	gsa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3802	iowa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3830	consumerfinance.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3889	login.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3933	fws.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4180	maine.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4322	medicare.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4402	cpsc.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4559	mn.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4773	delaware.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4855	energystar.gov	Blocks training only	allowed	allowed	allowed	blocked	allowed	allowed	allowed	allowed	–
5036	tsa.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5104	usaid.gov	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5419	doe.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5495	anl.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
5924	nv.gov	Blocks all AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
6394	ga.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6407	eeoc.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6419	war.gov	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
6423	dc.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6550	idaho.gov	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .gov

Government 145 sites

FAQ

Do .gov websites block AI crawlers?: As of 2026-10-02, 4.6% of the 108 .gov websites whose robots.txt we could read block GPTBot, 4.6% block ClaudeBot and 4.6% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 2 of them (1.9%) serve an llms.txt (all sites: 15.2%).
Which .gov websites block the most AI crawlers?: wa.gov (29 of 29), nv.gov (29 of 29), congress.gov (19 of 29), vermont.gov (7 of 29), ahrq.gov (3 of 29). The full per-site status is on each site page.
How is this measured?: We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 145 .gov sites are in the sample, 108 had a readable robots.txt (37 did not answer or returned an error page).

Other countries and suffixes

.com GPTBot 12.4%
.org GPTBot 11.3%
.net GPTBot 13.4%
.ru (Russia) GPTBot 5.1%
.de (Germany) GPTBot 23.6%
.jp (Japan) GPTBot 6%
.io GPTBot 4.3%
.edu GPTBot 3.7%
.br (Brazil) GPTBot 5.6%
.uk (United Kingdom) GPTBot 20%
.fr (France) GPTBot 31.8%
.in (India) GPTBot 4.1%
.cn (China) GPTBot 3.7%
.it (Italy) GPTBot 27.3%
.ai GPTBot 4.8%
.pl (Poland) GPTBot 26.4%
.au (Australia) GPTBot 27.8%
.tv GPTBot 6%
.nl (Netherlands) GPTBot 27.1%
.es (Spain) GPTBot 22.2%
.co GPTBot 9.1%
.ca (Canada) GPTBot 7.1%
.ch (Switzerland) GPTBot 16.7%
.ir (Iran) GPTBot 0%

Compare all suffixes