TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do education websites block AI crawlers?

As of 2026-10-01, 0% of the 20 education websites whose robots.txt we could read block GPTBot, 0% block ClaudeBot and 0% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 0 of 24 publish an llms.txt file.

Block rate by crawler: education vs all sites

CrawlerOperatorEducationAll sites
GPTBotOpenAI0%17%
OAI-SearchBotOpenAI0%6.7%
ChatGPT-UserOpenAI0%9.6%
OAI-AdsBotOpenAI0%4.3%
ClaudeBotAnthropic0%16.3%
Claude-SearchBotAnthropic0%8.1%
Claude-UserAnthropic0%8.3%
PerplexityBotPerplexity0%12.7%
Perplexity-UserPerplexity0%8.9%
Google-ExtendedGoogle0%14%
Google-CloudVertexBotGoogle0%7.2%
Google-GeminiNotebookGoogle0%4.3%
Google-AgentGoogle0%4.5%
ApplebotApple0%2.6%
Applebot-ExtendedApple0%14.3%
Meta-ExternalAgentMeta0%14.4%
meta-webindexerMeta0%6.3%
meta-externalfetcherMeta0%8.7%
meta-externaladsMeta0%3.8%
AmazonbotAmazon0%13.1%
Amzn-SearchBotAmazon0%5.9%
Amzn-UserAmazon0%5.5%
DuckAssistBotDuckDuckGo0%8.1%
MistralAI-UserMistral0%8.4%
MistralAI-IndexMistral0%4.7%
MistralAI-TrainingMistral0%4.7%
CCBotCommon Crawl0%19.5%
BytespiderByteDance0%17.7%
cohere-aiCohere0%13.7%

All 24 education sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
97mit.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
130harvard.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
166stanford.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
252cornell.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
267berkeley.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
402princeton.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
410washington.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
452ox.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
458umich.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
463columbia.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
523yale.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
542upenn.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
598psu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
646ucla.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
698wisc.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
699umn.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
721cam.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
746uci.eduNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
834cmu.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
861si.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
884uchicago.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
885ucsd.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
888utexas.eduOpenallowedallowedallowedallowedallowedallowedallowedallowed–
961jhu.eduUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

FAQ

Do education websites block AI crawlers?
As of 2026-10-01, 0% of the 20 education websites whose robots.txt we could read block GPTBot, 0% block ClaudeBot and 0% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 0 of 24 publish an llms.txt file.
Which education websites block GPTBot?
None in the index.

Other categories