TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · domain suffix

Do .org websites block AI crawlers?

As of 2026-10-03, 11.1% of the 514 .org websites whose robots.txt we could read block GPTBot, 6% block ClaudeBot and 3.9% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 44 of them (8.6%) serve an llms.txt (all sites: 15.2%).

11.1%
block GPTBot (all sites 12%)
6%
block ClaudeBot (all sites 11%)
8.6%
serve an llms.txt (all sites 15.2%)
514
of 587 sites with a readable robots.txt

Block rate by crawler: .org vs all sites

CrawlerOperator.orgAll sitesDifference
GPTBotOpenAI11.1%12%-0.9 pp
CCBotCommon Crawl9.1%12.8%-3.7 pp
BytespiderByteDance7.8%12.3%-4.5 pp
Google-ExtendedGoogle7.2%9.5%-2.3 pp
ClaudeBotAnthropic6%11%-5 pp
AmazonbotAmazon5.6%9.2%-3.6 pp
Applebot-ExtendedApple5.4%9.6%-4.2 pp
ChatGPT-UserOpenAI5.1%7.5%-2.4 pp
Meta-ExternalAgentMeta4.9%9.5%-4.6 pp
cohere-aiCohere4.7%9.1%-4.4 pp
PerplexityBotPerplexity3.9%8.3%-4.4 pp
Google-CloudVertexBotGoogle2.5%5.2%-2.7 pp
meta-externalfetcherMeta2.3%6%-3.7 pp
ApplebotApple2.1%3.1%-1 pp
Claude-SearchBotAnthropic1.9%5.4%-3.5 pp
MistralAI-TrainingMistral1.9%3.3%-1.4 pp
MistralAI-UserMistral1.9%5.4%-3.5 pp
OAI-SearchBotOpenAI1.9%5.7%-3.8 pp
Claude-UserAnthropic1.8%5.5%-3.7 pp
DuckAssistBotDuckDuckGo1.8%5.8%-4 pp
Perplexity-UserPerplexity1.8%5.7%-3.9 pp
meta-webindexerMeta1.6%4.6%-3 pp
Amzn-SearchBotAmazon1.2%4.1%-2.9 pp
Amzn-UserAmazon1.2%3.9%-2.7 pp
Google-AgentGoogle1.2%3.2%-2 pp
Google-GeminiNotebookGoogle1.2%3%-1.8 pp
meta-externaladsMeta1.2%2.8%-1.6 pp
MistralAI-IndexMistral1.2%3.3%-2.1 pp
OAI-AdsBotOpenAI1.2%3%-1.8 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open38866.1%
Unreadable7312.4%
No robots.txt539%
Blocks some AI search396.6%
Blocks training only325.5%
Blocks all AI search20.3%

Most restrictive .org sites

  1. openssl.org 29 of 29 blocked
  2. ebird.org 29 of 29 blocked
  3. tjk.org 28 of 29 blocked
  4. wto.org 28 of 29 blocked
  5. wikicv.org 28 of 29 blocked
  6. usgbc.org 28 of 29 blocked
  7. themoviedb.org 20 of 29 blocked
  8. tmdb.org 20 of 29 blocked
  9. gentoo.org 20 of 29 blocked
  10. wbur.org 13 of 29 blocked
  11. codeberg.org 12 of 29 blocked
  12. unep.org 11 of 29 blocked
  13. tvtropes.org 11 of 29 blocked
  14. poetryfoundation.org 11 of 29 blocked
  15. britishmuseum.org 11 of 29 blocked

Sites with an llms.txt (44)

Top 100 of 587 .org sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
16wikipedia.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
22wordpress.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
34nginx.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
42mozilla.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
56apache.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
64archive.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
75base.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
79w3.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
82creativecommons.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
92doi.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
103telegram.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
109wikimedia.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
132debian.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
147gnu.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
156un.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
203arxiv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
212ietf.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
248openstreetmap.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
270ieee.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
275unesco.orgBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblockedyes
301cambridge.orgBlocks some AI searchallowedallowedblockedallowedallowedallowedallowedallowed–
310iso.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
335python.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
351craigslist.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
390frontiersin.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
397hbr.orgBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
402worldbank.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
405mayoclinic.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
434change.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
440rfc-editor.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
453weforum.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
460spamhaus.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
461ecosia.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
479pbs.orgBlocks some AI searchblockedallowedallowedallowedallowedblockedallowedblocked–
483adblockplus.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
489archiveofourown.orgBlocks some AI searchblockedallowedblockedallowedallowedallowedallowedblocked–
492oecd.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
524hltv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
526apa.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
539plos.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
544jw.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
563coursera.orgBlocks training onlypartialpartialpartialpartialallowedpartialpartialpartialyes
566lichess.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
573wiktionary.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
604themoviedb.orgBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
610bbb.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
615postgresql.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
632jstor.orgBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
657aternos.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
664tgju.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
677kernel.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
705nosv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
728fedoraproject.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
749tjk.orgBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
767acm.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
803eff.orgOpenpartialallowedallowedpartialallowedallowedpartialpartial–
853fao.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
868acs.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
882moodle.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
884pewresearch.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
915clevelandclinic.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
922orcid.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
929sciencemag.orgBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
966pnas.orgBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
970mitre.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1008freebsd.orgOpenpartialallowedallowedallowedallowedallowedallowedpartial–
1019science.orgBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
1054bitbucket.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1077unicef.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1096aboutcookies.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1109openssl.orgBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1112nodejs.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1134semanticscholar.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1160khanacademy.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1181signal.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1182example.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1184kde.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1193altervista.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1194icann.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1219privacy-center.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1255imf.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1259opensource.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1261collegeboard.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1262nejm.orgBlocks some AI searchblockedallowedblockedallowedallowedblockedblockedblocked–
1264pypi.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1317apachefriends.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1347wikisource.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1350slashdot.orgOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1357perl.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1362cve.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1372matomo.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1375drupal.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1377tizen.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1395tmdb.orgBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1398gutenberg.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1411archlinux.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1434aarp.orgBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
1448centos.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1465ilo.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1499iop.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .org

FAQ

Do .org websites block AI crawlers?
As of 2026-10-03, 11.1% of the 514 .org websites whose robots.txt we could read block GPTBot, 6% block ClaudeBot and 3.9% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 44 of them (8.6%) serve an llms.txt (all sites: 15.2%).
Which .org websites block the most AI crawlers?
openssl.org (29 of 29), ebird.org (29 of 29), tjk.org (28 of 29), wto.org (28 of 29), wikicv.org (28 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 587 .org sites are in the sample, 514 had a readable robots.txt (73 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes