AI Crawler Index · category
Do other popular websites block AI crawlers?
As of 2026-11-16, 15.1% of the 6,651 other popular websites whose robots.txt we could read block GPTBot, 14.1% block ClaudeBot and 7.8% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1185 of 7963 publish an llms.txt file.
Block rate by crawler: other popular vs all sites
| Crawler | Operator | Other | All sites |
|---|---|---|---|
| CCBot | Common Crawl | 16% | 16.4% |
| Bytespider | ByteDance | 15.6% | 16.2% |
| GPTBot | OpenAI | 15.1% | 15.4% |
| ClaudeBot | Anthropic | 14.1% | 14.7% |
| Meta-ExternalAgent | Meta | 13.3% | 13.6% |
| Google-Extended | 13% | 13.2% | |
| Applebot-Extended | Apple | 12.9% | 13.4% |
| Amazonbot | Amazon | 12.9% | 13.2% |
| cohere-ai | Cohere | 12.3% | 12.7% |
| Google-CloudVertexBot | 8.9% | 8.8% | |
| PerplexityBot | Perplexity | 7.8% | 8.2% |
| ChatGPT-User | OpenAI | 7.2% | 7.4% |
| MistralAI-Training | Mistral | 7.1% | 7.1% |
| meta-externalfetcher | Meta | 6.2% | 6.5% |
| Perplexity-User | Perplexity | 5.7% | 6.1% |
| DuckAssistBot | DuckDuckGo | 5.7% | 5.9% |
| OAI-SearchBot | OpenAI | 5.6% | 5.8% |
| Claude-User | Anthropic | 5.6% | 5.9% |
| MistralAI-User | Mistral | 5.6% | 5.7% |
| Claude-SearchBot | Anthropic | 5.3% | 5.7% |
| meta-webindexer | Meta | 5% | 5.1% |
| Amzn-User | Amazon | 4.5% | 4.4% |
| Amzn-SearchBot | Amazon | 4.4% | 4.4% |
| Google-Agent | 3.6% | 3.6% | |
| MistralAI-Index | Mistral | 3.6% | 3.6% |
| Google-GeminiNotebook | 3.4% | 3.3% | |
| Applebot | Apple | 3.3% | 3.2% |
| OAI-AdsBot | OpenAI | 3% | 3% |
| meta-externalads | Meta | 3% | 2.9% |
Top 150 of 7,963 other popular sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 10 | dzen.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 14 | azure.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 18 | cloud.microsoft | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 20 | gandi.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 45 | dellemc.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 47 | windows.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 54 | reg.ru | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 75 | base.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 76 | macromedia.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 77 | hosting24.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 80 | flickr.com | Blocks some AI search | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 82 | creativecommons.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 113 | roku.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 117 | launchpad.net | Blocks some AI search | blocked | blocked | allowed | blocked | blocked | blocked | blocked | blocked | – |
| 119 | mobile.de | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 123 | googleblog.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 138 | drom.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 139 | crpt.ru | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 142 | aliyun.com | Open | partial | allowed | allowed | partial | allowed | allowed | allowed | partial | – |
| 143 | name.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 151 | trustpilot.com | Blocks some AI search | partial | partial | partial | partial | partial | partial | partial | partial | – |
| 155 | selectel.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 167 | netangels.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 175 | indeed.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 179 | timeweb.ru | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 182 | go.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | blocked | allowed | – |
| 206 | amazonalexa.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 213 | cloudflareinsights.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 219 | bitdefender.net | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 220 | gosuslugi.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 231 | eventbrite.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 233 | share.google | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 236 | bitrix24.ru | Open | partial | partial | partial | partial | partial | partial | partial | allowed | yes |
| 237 | capcut.com | Open | partial | partial | partial | partial | partial | partial | partial | partial | yes |
| 241 | blogger.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 256 | oraclecloud.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 271 | ovh.net | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 298 | yelp.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 304 | vkontakte.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 305 | porkbun.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 308 | wpguardian.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 309 | consultant.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 311 | uk.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 314 | outbrain.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 320 | fb.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 321 | oxylabs.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 331 | webempresa.eu | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 337 | beget.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 367 | discordapp.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 373 | usps.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 376 | pccc.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 385 | pendo.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 387 | life360.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 399 | it.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 406 | agora.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 418 | deloitte.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 421 | smartthings.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 457 | eu.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 472 | kontur.ru | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 504 | safety.google | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 514 | lge.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 521 | superhosting.bg | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 535 | inven.co.kr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 536 | one.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 541 | locaweb.com.br | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 546 | inleed.net | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 565 | tabor.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 572 | hostpoint.ch | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 588 | us.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 591 | 0xrpc.io | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 593 | jusbrasil.com.br | Blocks some AI search | partial | partial | partial | partial | partial | partial | partial | blocked | yes |
| 598 | beboo.ru | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 600 | ap4r.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 602 | litnet.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 609 | naukri.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 610 | bbb.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 611 | id.me | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 623 | grow.google | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 627 | labs.google | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 628 | edna.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 635 | jino.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 661 | bookmyshow.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 665 | mbga.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 686 | sotwe.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 690 | home.pl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 695 | cmoa.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 696 | platform.sh | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 701 | max.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 705 | nosv.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 723 | testbook.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 727 | 1mg.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 729 | immobiliare.it | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 733 | notion.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 734 | manulife-sinochem.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 735 | absher.sa | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 736 | bt.dk | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 737 | xhwide.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 738 | programme-tv.net | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | yes |
| 740 | aajtak.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 742 | iau.ir | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 743 | hover.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 744 | goodreturns.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 745 | kakuyomu.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 747 | ppomppu.co.kr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 748 | deezer.com | Open | partial | partial | partial | partial | partial | partial | partial | partial | – |
| 749 | tjk.org | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 751 | rakuten-sec.co.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 754 | bikewale.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 756 | trilltrill.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 757 | aka.ms | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 761 | therapservices.net | Open | partial | partial | partial | partial | allowed | partial | partial | partial | yes |
| 762 | sinoptik.ua | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 763 | buzzfeed.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | blocked | blocked | – |
| 764 | wordwall.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 765 | autoplius.lt | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 766 | theqoo.net | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 767 | acm.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 768 | mongodb.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 769 | screener.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 770 | mamastar.jp | Blocks training only | allowed | allowed | allowed | partial | allowed | allowed | allowed | allowed | – |
| 772 | chordtela.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 773 | tupi.fm | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 774 | mechacomic.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 775 | girlschannel.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 776 | czbooks.net | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 777 | freewebnovel.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 778 | ss.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 780 | site.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 783 | skelbiu.lt | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 785 | rayjump.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 787 | eepurl.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 788 | smadex.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 789 | piccoma.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 790 | alphapolis.co.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 791 | lecturas.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 792 | rambler.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 793 | wpastra.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 794 | letras.mus.br | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 795 | publi24.ro | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 796 | afribaba.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 797 | animanch.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 801 | twkan.com | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | allowed | blocked | blocked | – |
| 803 | eff.org | Open | partial | allowed | allowed | partial | allowed | allowed | partial | partial | yes |
| 804 | dto.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 805 | arbada.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 808 | goal7.co | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 809 | ecnavi.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 812 | promiedos.com.ar | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 813 | tjsp.jus.br | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 815 | turbo.az | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.
FAQ
- Do other popular websites block AI crawlers?
- As of 2026-11-16, 15.1% of the 6,651 other popular websites whose robots.txt we could read block GPTBot, 14.1% block ClaudeBot and 7.8% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 1185 of 7963 publish an llms.txt file.
- Which other popular websites block GPTBot?
- flickr.com, launchpad.net, go.com, fb.com, litnet.com, bt.dk, tjk.org, aka.ms, buzzfeed.com, theqoo.net, and more (full list on this page).
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%