Which .edu websites block the most AI crawlers?

osu.edu (29 of 29), cuny.edu (28 of 29), academia.edu (21 of 29), ucsc.edu (8 of 29), nap.edu (7 of 29). The full per-site status is on each site page.

How is this measured?

We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 183 .edu sites are in the sample, 161 had a readable robots.txt (22 did not answer or returned an error page).

AI Crawler Index · domain suffix

Do .edu websites block AI crawlers?

As of 2026-10-03, 3.7% of the 161 .edu websites whose robots.txt we could read block GPTBot, 3.1% block ClaudeBot and 2.5% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 7 of them (4.3%) serve an llms.txt (all sites: 15.2%).

3.7%

block GPTBot (all sites 12%)

3.1%

block ClaudeBot (all sites 11%)

4.3%

serve an llms.txt (all sites 15.2%)

161

of 183 sites with a readable robots.txt

Block rate by crawler: .edu vs all sites

Crawler	Operator	.edu	All sites	Difference
Bytespider	ByteDance	5.6%	12.3%	-6.7 pp
Google-Extended	Google	3.7%	9.5%	-5.8 pp
GPTBot	OpenAI	3.7%	12%	-8.3 pp
CCBot	Common Crawl	3.1%	12.8%	-9.7 pp
ClaudeBot	Anthropic	3.1%	11%	-7.9 pp
Amazonbot	Amazon	2.5%	9.2%	-6.7 pp
Applebot-Extended	Apple	2.5%	9.6%	-7.1 pp
cohere-ai	Cohere	2.5%	9.1%	-6.6 pp
Meta-ExternalAgent	Meta	2.5%	9.5%	-7 pp
Perplexity-User	Perplexity	2.5%	5.7%	-3.2 pp
PerplexityBot	Perplexity	2.5%	8.3%	-5.8 pp
Amzn-SearchBot	Amazon	1.9%	4.1%	-2.2 pp
Amzn-User	Amazon	1.9%	3.9%	-2 pp
ChatGPT-User	OpenAI	1.9%	7.5%	-5.6 pp
Google-Agent	Google	1.9%	3.2%	-1.3 pp
Google-CloudVertexBot	Google	1.9%	5.2%	-3.3 pp
Google-GeminiNotebook	Google	1.9%	3%	-1.1 pp
meta-externalads	Meta	1.9%	2.8%	-0.9 pp
meta-externalfetcher	Meta	1.9%	6%	-4.1 pp
meta-webindexer	Meta	1.9%	4.6%	-2.7 pp
MistralAI-Index	Mistral	1.9%	3.3%	-1.4 pp
MistralAI-Training	Mistral	1.9%	3.3%	-1.4 pp
OAI-AdsBot	OpenAI	1.9%	3%	-1.1 pp
Claude-SearchBot	Anthropic	1.2%	5.4%	-4.2 pp
Claude-User	Anthropic	1.2%	5.5%	-4.3 pp
DuckAssistBot	DuckDuckGo	1.2%	5.8%	-4.6 pp
MistralAI-User	Mistral	1.2%	5.4%	-4.2 pp
OAI-SearchBot	OpenAI	1.2%	5.7%	-4.5 pp
Applebot	Apple	0.6%	3.1%	-2.5 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended.

Policy mix

Policy	Sites	Share
Open	137	74.9%
Unreadable	22	12%
No robots.txt	13	7.1%
Blocks some AI search	5	2.7%
Blocks training only	5	2.7%
Blocks all AI search	1	0.5%

Sites with an llms.txt (7)

Top 100 of 183 .edu sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#	Site	Policy	GPTBot	OAI-Search	ChatGPT-User	ClaudeBot	Claude-Search	Perplexity	Google-Ext.	CCBot	llms.txt
98	mit.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
133	harvard.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
169	stanford.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
257	cornell.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
273	berkeley.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
378	academia.edu	Blocks some AI search	blocked	partial	partial	blocked	partial	partial	blocked	blocked	–
404	princeton.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
415	washington.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
459	umich.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
466	columbia.edu	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
533	yale.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
550	upenn.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
606	psu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
649	ucla.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
706	wisc.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
711	umn.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
752	uci.edu	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
847	cmu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
877	si.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
902	ucsd.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
906	uchicago.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
911	utexas.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1005	jhu.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1081	asu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1084	ufl.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1136	duke.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1140	uiuc.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1146	purdue.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1165	msu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1168	nyu.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1174	unc.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1208	umd.edu	Blocks some AI search	blocked	allowed	allowed	allowed	allowed	blocked	blocked	allowed	–
1251	usc.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1268	ucdavis.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1301	arizona.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1313	illinois.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1337	colorado.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1376	northwestern.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1452	virginia.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1474	utah.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1504	tamu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1530	osu.edu	Blocks all AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
1567	bu.edu	Open	partial	allowed	allowed	partial	allowed	allowed	allowed	partial	–
1584	ucsb.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1624	ncsu.edu	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1653	rutgers.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1670	pitt.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1713	vt.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1733	caltech.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1734	brookings.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1767	monash.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1809	indiana.edu	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1829	georgetown.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1841	cuny.edu	Blocks some AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
1869	hawaii.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1870	wustl.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1911	tufts.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1960	gatech.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1997	uga.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2021	fsu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2023	unl.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2029	brown.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2038	iastate.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2050	hbs.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2065	rochester.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2092	oregonstate.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2148	uiowa.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2249	umass.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2303	iu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2361	usf.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2396	gwu.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2400	ucsf.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2425	udel.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2463	uoregon.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2474	vanderbilt.edu	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2590	nd.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2594	wsu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2636	colostate.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2661	byu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2665	rice.edu	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2711	ucsc.edu	Blocks training only	blocked	allowed	allowed	blocked	allowed	allowed	blocked	blocked	–
2712	gmu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2865	buffalo.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2899	clemson.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2915	uky.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2938	uw.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2960	dartmouth.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2973	ucr.edu	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2976	emory.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3081	miami.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3127	uh.edu	Blocks training only	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3155	lsu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3169	missouri.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3201	uconn.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3540	ku.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3608	utk.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3638	unm.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3792	rit.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3910	gsu.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3958	ucar.edu	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .edu

Education 182 sites
Reference 1 sites

FAQ

Do .edu websites block AI crawlers?: As of 2026-10-03, 3.7% of the 161 .edu websites whose robots.txt we could read block GPTBot, 3.1% block ClaudeBot and 2.5% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 7 of them (4.3%) serve an llms.txt (all sites: 15.2%).
Which .edu websites block the most AI crawlers?: osu.edu (29 of 29), cuny.edu (28 of 29), academia.edu (21 of 29), ucsc.edu (8 of 29), nap.edu (7 of 29). The full per-site status is on each site page.
How is this measured?: We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 183 .edu sites are in the sample, 161 had a readable robots.txt (22 did not answer or returned an error page).

Other countries and suffixes

.com GPTBot 12.4%
.org GPTBot 11.1%
.net GPTBot 13.6%
.ru (Russia) GPTBot 5.1%
.de (Germany) GPTBot 23.6%
.jp (Japan) GPTBot 6%
.io GPTBot 4.3%
.gov GPTBot 4.6%
.br (Brazil) GPTBot 6.7%
.uk (United Kingdom) GPTBot 20%
.fr (France) GPTBot 31.8%
.in (India) GPTBot 4.1%
.cn (China) GPTBot 3.8%
.it (Italy) GPTBot 27.3%
.ai GPTBot 4.9%
.pl (Poland) GPTBot 26.4%
.au (Australia) GPTBot 27%
.tv GPTBot 6%
.nl (Netherlands) GPTBot 27.1%
.es (Spain) GPTBot 25%
.co GPTBot 8.9%
.ca (Canada) GPTBot 7.1%
.ch (Switzerland) GPTBot 16.7%
.ir (Iran) GPTBot 0%

Compare all suffixes