AI Crawler Index · category
Do education websites block AI crawlers?
As of 2026-11-16, 5.6% of the 232 education websites whose robots.txt we could read block GPTBot, 4.7% block ClaudeBot and 2.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 14 of 289 publish an llms.txt file.
Block rate by crawler: education vs all sites
| Crawler | Operator | Education | All sites |
|---|---|---|---|
| Bytespider | ByteDance | 6.9% | 16.2% |
| GPTBot | OpenAI | 5.6% | 15.4% |
| ClaudeBot | Anthropic | 4.7% | 14.7% |
| Amazonbot | Amazon | 4.7% | 13.2% |
| CCBot | Common Crawl | 4.7% | 16.4% |
| Google-Extended | 4.3% | 13.2% | |
| Google-CloudVertexBot | 3.9% | 8.8% | |
| Applebot-Extended | Apple | 3.9% | 13.4% |
| Meta-ExternalAgent | Meta | 3.9% | 13.6% |
| cohere-ai | Cohere | 3.9% | 12.7% |
| MistralAI-Training | Mistral | 3.4% | 7.1% |
| PerplexityBot | Perplexity | 2.2% | 8.2% |
| ChatGPT-User | OpenAI | 1.7% | 7.4% |
| Perplexity-User | Perplexity | 1.7% | 6.1% |
| OAI-SearchBot | OpenAI | 1.3% | 5.8% |
| Amzn-SearchBot | Amazon | 1.3% | 4.4% |
| Amzn-User | Amazon | 1.3% | 4.4% |
| OAI-AdsBot | OpenAI | 0.9% | 3% |
| Claude-SearchBot | Anthropic | 0.9% | 5.7% |
| Claude-User | Anthropic | 0.9% | 5.9% |
| Google-GeminiNotebook | 0.9% | 3.3% | |
| Google-Agent | 0.9% | 3.6% | |
| meta-webindexer | Meta | 0.9% | 5.1% |
| meta-externalfetcher | Meta | 0.9% | 6.5% |
| meta-externalads | Meta | 0.9% | 2.9% |
| DuckAssistBot | DuckDuckGo | 0.9% | 5.9% |
| MistralAI-User | Mistral | 0.9% | 5.7% |
| MistralAI-Index | Mistral | 0.9% | 3.6% |
| Applebot | Apple | 0.4% | 3.2% |
Top 150 of 289 education sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 98 | mit.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 133 | harvard.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 169 | stanford.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 257 | cornell.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 273 | berkeley.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 404 | princeton.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 415 | washington.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 458 | ox.ac.uk | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 459 | umich.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 466 | columbia.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 533 | yale.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 550 | upenn.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 606 | psu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 649 | ucla.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 706 | wisc.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 711 | umn.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 731 | cam.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 752 | uci.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 847 | cmu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 877 | si.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 902 | ucsd.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 906 | uchicago.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 911 | utexas.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1005 | jhu.edu | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1081 | asu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1084 | ufl.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1136 | duke.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1140 | uiuc.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1146 | purdue.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1165 | msu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 1168 | nyu.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1174 | unc.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1208 | umd.edu | Blocks some AI search | blocked | allowed | allowed | allowed | allowed | blocked | allowed | allowed | – |
| 1251 | usc.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1268 | ucdavis.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1301 | arizona.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1313 | illinois.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1337 | colorado.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1348 | ucl.ac.uk | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1376 | northwestern.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1388 | sjtu.edu.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1452 | virginia.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1456 | tsinghua.edu.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1474 | utah.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1504 | tamu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1528 | univie.ac.at | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1530 | osu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1567 | bu.edu | Open | partial | allowed | allowed | partial | allowed | allowed | allowed | partial | – |
| 1584 | ucsb.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1605 | unimelb.edu.au | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1624 | ncsu.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1653 | rutgers.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1670 | pitt.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1713 | vt.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1733 | caltech.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1734 | brookings.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1767 | monash.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1809 | indiana.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1829 | georgetown.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1841 | cuny.edu | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 1869 | hawaii.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1870 | wustl.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1911 | tufts.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1918 | lse.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1960 | gatech.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1997 | uga.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2021 | fsu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2023 | unl.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2029 | brown.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2038 | iastate.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2050 | hbs.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2065 | rochester.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2079 | anu.edu.au | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2092 | oregonstate.edu | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 2148 | uiowa.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2249 | umass.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2281 | unsw.edu.au | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2303 | iu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2361 | usf.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2396 | gwu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2400 | ucsf.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2411 | u-tokyo.ac.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2425 | udel.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2463 | uoregon.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2474 | vanderbilt.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2514 | pku.edu.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2590 | nd.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2594 | wsu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2618 | uq.edu.au | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2636 | colostate.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2661 | byu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2665 | rice.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2711 | ucsc.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2712 | gmu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2804 | sydney.edu.au | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2818 | manchester.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2865 | buffalo.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2899 | clemson.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2915 | uky.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2938 | uw.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2960 | dartmouth.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2961 | gla.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2973 | ucr.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2976 | emory.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3075 | nus.edu.sg | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3081 | miami.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3105 | imperial.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3127 | uh.edu | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3155 | lsu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3169 | missouri.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3177 | warwick.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3201 | uconn.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3207 | kcl.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3231 | cuhk.edu.hk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3471 | leeds.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3540 | ku.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3608 | utk.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3638 | unm.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3640 | icm.edu.pl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3792 | rit.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3851 | kyoto-u.ac.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3892 | adelaide.edu.au | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3910 | gsu.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3958 | ucar.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3961 | ub.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3971 | fiu.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4147 | getty.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4156 | unh.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4269 | york.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4271 | wvu.edu | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4292 | northeastern.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4323 | syr.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4326 | ncl.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4340 | bc.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4445 | nii.ac.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4581 | ou.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4841 | unt.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4853 | nptel.ac.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4935 | temple.edu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5008 | rpi.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5080 | ucf.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5092 | unisa.ac.za | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5097 | uvm.edu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5098 | aiou.edu.pk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5171 | uwa.edu.au | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5231 | newcastle.edu.au | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5294 | samarth.edu.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5303 | nap.edu | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 5347 | st-andrews.ac.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 5383 | nottingham.ac.uk | Blocks some AI search | blocked | blocked | blocked | blocked | allowed | blocked | blocked | blocked | – |
Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.
FAQ
- Do education websites block AI crawlers?
- As of 2026-11-16, 5.6% of the 232 education websites whose robots.txt we could read block GPTBot, 4.7% block ClaudeBot and 2.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 14 of 289 publish an llms.txt file.
- Which education websites block GPTBot?
- jhu.edu, umd.edu, cuny.edu, oregonstate.edu, nap.edu, nottingham.ac.uk, nhm.ac.uk, uct.ac.za, bisefsd.edu.pk, snu.ac.kr, and more (full list on this page).
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%