TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do education websites block AI crawlers?

As of 2026-11-16, 5.6% of the 232 education websites whose robots.txt we could read block GPTBot, 4.7% block ClaudeBot and 2.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 14 of 289 publish an llms.txt file.

Block rate by crawler: education vs all sites

CrawlerOperatorEducationAll sites
BytespiderByteDance6.9%16.2%
GPTBotOpenAI5.6%15.4%
ClaudeBotAnthropic4.7%14.7%
AmazonbotAmazon4.7%13.2%
CCBotCommon Crawl4.7%16.4%
Google-ExtendedGoogle4.3%13.2%
Google-CloudVertexBotGoogle3.9%8.8%
Applebot-ExtendedApple3.9%13.4%
Meta-ExternalAgentMeta3.9%13.6%
cohere-aiCohere3.9%12.7%
MistralAI-TrainingMistral3.4%7.1%
PerplexityBotPerplexity2.2%8.2%
ChatGPT-UserOpenAI1.7%7.4%
Perplexity-UserPerplexity1.7%6.1%
OAI-SearchBotOpenAI1.3%5.8%
Amzn-SearchBotAmazon1.3%4.4%
Amzn-UserAmazon1.3%4.4%
OAI-AdsBotOpenAI0.9%3%
Claude-SearchBotAnthropic0.9%5.7%
Claude-UserAnthropic0.9%5.9%
Google-GeminiNotebookGoogle0.9%3.3%
Google-AgentGoogle0.9%3.6%
meta-webindexerMeta0.9%5.1%
meta-externalfetcherMeta0.9%6.5%
meta-externaladsMeta0.9%2.9%
DuckAssistBotDuckDuckGo0.9%5.9%
MistralAI-UserMistral0.9%5.7%
MistralAI-IndexMistral0.9%3.6%
ApplebotApple0.4%3.2%

Top 150 of 289 education sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
98mit.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
133harvard.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
169stanford.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
257cornell.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
273berkeley.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
404princeton.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
415washington.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
458ox.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
459umich.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
466columbia.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
533yale.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
550upenn.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
606psu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
649ucla.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
706wisc.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
711umn.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
731cam.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
752uci.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
847cmu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
877si.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
902ucsd.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
906uchicago.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
911utexas.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1005jhu.eduBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
1081asu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1084ufl.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1136duke.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1140uiuc.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1146purdue.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1165msu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1168nyu.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1174unc.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1208umd.eduBlocks some AI searchblockedallowedallowedallowedallowedblockedallowedallowed–
1251usc.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1268ucdavis.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1301arizona.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1313illinois.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1337colorado.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1348ucl.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1376northwestern.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1388sjtu.edu.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1452virginia.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1456tsinghua.edu.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1474utah.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1504tamu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1528univie.ac.atOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1530osu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1567bu.eduOpenpartialallowedallowedpartialallowedallowedallowedpartial–
1584ucsb.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1605unimelb.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1624ncsu.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1653rutgers.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1670pitt.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1713vt.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1733caltech.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1734brookings.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1767monash.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1809indiana.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1829georgetown.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1841cuny.eduBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1869hawaii.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1870wustl.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1911tufts.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1918lse.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1960gatech.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1997uga.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2021fsu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2023unl.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2029brown.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2038iastate.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2050hbs.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2065rochester.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2079anu.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2092oregonstate.eduBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
2148uiowa.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2249umass.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2281unsw.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2303iu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2361usf.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2396gwu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2400ucsf.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2411u-tokyo.ac.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2425udel.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2463uoregon.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2474vanderbilt.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2514pku.edu.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2590nd.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2594wsu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2618uq.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2636colostate.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2661byu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2665rice.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2711ucsc.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2712gmu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2804sydney.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2818manchester.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2865buffalo.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2899clemson.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2915uky.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2938uw.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2960dartmouth.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2961gla.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2973ucr.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2976emory.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3075nus.edu.sgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3081miami.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3105imperial.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3127uh.eduBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
3155lsu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3169missouri.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3177warwick.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3201uconn.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3207kcl.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3231cuhk.edu.hkOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3471leeds.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3540ku.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3608utk.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3638unm.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3640icm.edu.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3792rit.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3851kyoto-u.ac.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3892adelaide.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3910gsu.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3958ucar.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3961ub.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3971fiu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4147getty.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4156unh.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4269york.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4271wvu.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4292northeastern.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4323syr.eduOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4326ncl.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4340bc.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4445nii.ac.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4581ou.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4841unt.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4853nptel.ac.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4935temple.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
5008rpi.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5080ucf.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5092unisa.ac.zaUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5097uvm.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5098aiou.edu.pkOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5171uwa.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5231newcastle.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5294samarth.edu.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5303nap.eduBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
5347st-andrews.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
5383nottingham.ac.ukBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–

Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.

FAQ

Do education websites block AI crawlers?
As of 2026-11-16, 5.6% of the 232 education websites whose robots.txt we could read block GPTBot, 4.7% block ClaudeBot and 2.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 14 of 289 publish an llms.txt file.
Which education websites block GPTBot?
jhu.edu, umd.edu, cuny.edu, oregonstate.edu, nap.edu, nottingham.ac.uk, nhm.ac.uk, uct.ac.za, bisefsd.edu.pk, snu.ac.kr, and more (full list on this page).

Other categories