AI Crawler Index · domain suffix
Do .com websites block AI crawlers?
As of 2026-11-16, 15.9% of the 4,449 .com websites whose robots.txt we could read block GPTBot, 15.7% block ClaudeBot and 8.9% block PerplexityBot (all 8,313 sites: 15.4%, 14.7%, 8.2%). 941 of them (21.2%) serve an llms.txt (all sites: 16.7%).
15.9%
block GPTBot (all sites 15.4%)
15.7%
block ClaudeBot (all sites 14.7%)
21.2%
serve an llms.txt (all sites 16.7%)
4,449
of 5,298 sites with a readable robots.txt
Block rate by crawler: .com vs all sites
| Crawler | Operator | .com | All sites | Difference |
|---|---|---|---|---|
| Bytespider | ByteDance | 18.1% | 16.2% | +1.9 pp |
| CCBot | Common Crawl | 17.8% | 16.4% | +1.4 pp |
| GPTBot | OpenAI | 15.9% | 15.4% | +0.5 pp |
| ClaudeBot | Anthropic | 15.7% | 14.7% | +1 pp |
| Meta-ExternalAgent | Meta | 14.9% | 13.6% | +1.3 pp |
| Applebot-Extended | Apple | 14.5% | 13.4% | +1.1 pp |
| Amazonbot | Amazon | 14.2% | 13.2% | +1 pp |
| Google-Extended | 14% | 13.2% | +0.8 pp | |
| cohere-ai | Cohere | 13.4% | 12.7% | +0.7 pp |
| Google-CloudVertexBot | 9.4% | 8.8% | +0.6 pp | |
| PerplexityBot | Perplexity | 8.9% | 8.2% | +0.7 pp |
| ChatGPT-User | OpenAI | 7.6% | 7.4% | +0.2 pp |
| meta-externalfetcher | Meta | 7.3% | 6.5% | +0.8 pp |
| MistralAI-Training | Mistral | 7.2% | 7.1% | +0.1 pp |
| Perplexity-User | Perplexity | 7% | 6.1% | +0.9 pp |
| Claude-User | Anthropic | 6.7% | 5.9% | +0.8 pp |
| DuckAssistBot | DuckDuckGo | 6.6% | 5.9% | +0.7 pp |
| MistralAI-User | Mistral | 6.4% | 5.7% | +0.7 pp |
| Claude-SearchBot | Anthropic | 6.3% | 5.7% | +0.6 pp |
| meta-webindexer | Meta | 6.2% | 5.1% | +1.1 pp |
| OAI-SearchBot | OpenAI | 5.7% | 5.8% | -0.1 pp |
| Amzn-SearchBot | Amazon | 5.3% | 4.4% | +0.9 pp |
| Amzn-User | Amazon | 5.1% | 4.4% | +0.7 pp |
| MistralAI-Index | Mistral | 4% | 3.6% | +0.4 pp |
| Google-Agent | 3.9% | 3.6% | +0.3 pp | |
| Google-GeminiNotebook | 3.6% | 3.3% | +0.3 pp | |
| OAI-AdsBot | OpenAI | 3.2% | 3% | +0.2 pp |
| Applebot | Apple | 3.1% | 3.2% | -0.1 pp |
| meta-externalads | Meta | 3.1% | 2.9% | +0.2 pp |
Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.
Policy mix
| Policy | Sites | Share |
|---|---|---|
| Open | 2,918 | 55.1% |
| Unreadable | 849 | 16% |
| Blocks some AI search | 613 | 11.6% |
| No robots.txt | 518 | 9.8% |
| Blocks training only | 331 | 6.2% |
| Blocks all AI search | 69 | 1.3% |
Most restrictive .com sites
- reddit.com 29 of 29 blocked
- qcloud.com 29 of 29 blocked
- adrta.com 29 of 29 blocked
- constellix.com 29 of 29 blocked
- kasserver.com 29 of 29 blocked
- co.com 29 of 29 blocked
- tappx.com 29 of 29 blocked
- urldefense.com 29 of 29 blocked
- system-monitor.com 29 of 29 blocked
- jwpltx.com 29 of 29 blocked
- hellotalk8.com 29 of 29 blocked
- lenovomm.com 29 of 29 blocked
- onenote.com 29 of 29 blocked
- sportradarserving.com 29 of 29 blocked
- backblazeb2.com 29 of 29 blocked
Sites with an llms.txt (941)
Top 100 of 5,298 .com sites
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | google.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2 | cloudflare.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3 | facebook.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 4 | microsoft.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5 | youtube.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6 | apple.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7 | instagram.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 9 | twitter.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 11 | linkedin.com | Blocks some AI search | blocked | partial | blocked | blocked | partial | blocked | blocked | blocked | – |
| 12 | office.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 13 | amazon.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 14 | azure.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 15 | github.com | Blocks training only | partial | partial | partial | partial | partial | partial | allowed | allowed | yes |
| 17 | bing.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 19 | netflix.com | Blocks some AI search | partial | partial | partial | partial | partial | partial | partial | blocked | – |
| 21 | x.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 23 | skype.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 24 | icloud.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 25 | pinterest.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 26 | tiktok.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 27 | whatsapp.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | blocked | allowed | – |
| 28 | yahoo.com | Blocks some AI search | blocked | partial | blocked | blocked | partial | blocked | blocked | blocked | – |
| 29 | roblox.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 30 | adobe.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 31 | spotify.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 33 | qq.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 35 | baidu.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 36 | chatgpt.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | blocked | blocked | blocked | – |
| 37 | opera.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 38 | vimeo.com | Blocks training only | partial | allowed | allowed | partial | allowed | allowed | partial | blocked | – |
| 39 | openai.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 40 | f5.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 45 | dellemc.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 46 | samsung.com | Open | partial | partial | partial | allowed | allowed | partial | partial | allowed | yes |
| 47 | windows.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 48 | reddit.com | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 49 | ui.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 50 | office365.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 52 | amazonvideo.com | Blocks some AI search | allowed | allowed | blocked | allowed | allowed | blocked | allowed | blocked | – |
| 53 | epicgames.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 55 | snapchat.com | Blocks some AI search | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 57 | dropbox.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 58 | intuit.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 59 | vk.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 62 | unity3d.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 63 | kaspersky.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 65 | godaddy.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 66 | tumblr.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 67 | nytimes.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 68 | xiaomi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 70 | webex.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | yes |
| 71 | paypal.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 72 | shopify.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 73 | spectrum.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 76 | macromedia.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 77 | hosting24.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 78 | paytmfirstgames.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 80 | flickr.com | Blocks some AI search | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 81 | soundcloud.com | Open | partial | partial | partial | partial | partial | partial | partial | partial | – |
| 83 | medium.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | allowed | – |
| 84 | cnn.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 85 | theguardian.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | allowed | blocked | – |
| 86 | ebay.com | Blocks some AI search | blocked | partial | partial | blocked | partial | blocked | allowed | blocked | – |
| 87 | android.com | Open | partial | partial | allowed | partial | partial | partial | partial | partial | – |
| 91 | oracle.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 93 | ubuntu.com | Open | partial | partial | partial | partial | partial | partial | partial | allowed | yes |
| 94 | forbes.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | allowed | blocked | – |
| 95 | bbc.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 96 | sciencedirect.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 100 | cisco.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 104 | stripe.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 106 | canva.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 110 | meraki.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 111 | gmail.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 112 | hubspot.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 113 | roku.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 115 | imdb.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 116 | namecheap.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 118 | booking.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 120 | ibm.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 121 | reuters.com | Blocks some AI search | blocked | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 122 | springer.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 123 | googleblog.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 124 | hp.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 125 | autodesk.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 126 | nature.com | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 128 | steampowered.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 129 | weibo.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 130 | avast.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 134 | alibaba.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | blocked | allowed | – |
| 135 | weather.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | yes |
| 136 | wiley.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 137 | dell.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 141 | etsy.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 142 | aliyun.com | Open | partial | allowed | allowed | partial | allowed | allowed | allowed | partial | – |
| 143 | name.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 144 | mi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 145 | weebly.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 146 | bilibili.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 148 | arubanetworks.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
Sorted by popularity; the rest are on the A-Z lists.
Categories on .com
- Other 4,470 sites
- News & media 250 sites
- Entertainment 180 sites
- Tech 142 sites
- Finance 87 sites
- E-commerce 81 sites
- Social 38 sites
- Reference 28 sites
- Travel 11 sites
- Telecom 6 sites
- AI 5 sites
FAQ
- Do .com websites block AI crawlers?
- As of 2026-11-16, 15.9% of the 4,449 .com websites whose robots.txt we could read block GPTBot, 15.7% block ClaudeBot and 8.9% block PerplexityBot (all 8,313 sites: 15.4%, 14.7%, 8.2%). 941 of them (21.2%) serve an llms.txt (all sites: 16.7%).
- Which .com websites block the most AI crawlers?
- reddit.com (29 of 29), qcloud.com (29 of 29), adrta.com (29 of 29), constellix.com (29 of 29), kasserver.com (29 of 29). The full per-site status is on each site page.
- How is this measured?
- We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 5,298 .com sites are in the sample, 4,449 had a readable robots.txt (849 did not answer or returned an error page).
Other countries and suffixes
- .org GPTBot 14.8%
- .net GPTBot 17.8%
- .ru (Russia) GPTBot 4.9%
- .de (Germany) GPTBot 24.9%
- .jp (Japan) GPTBot 6.9%
- .io GPTBot 7%
- .edu GPTBot 4.4%
- .gov GPTBot 4.7%
- .br (Brazil) GPTBot 8.1%
- .uk (United Kingdom) GPTBot 22%
- .fr (France) GPTBot 34.1%
- .in (India) GPTBot 5.6%
- .cn (China) GPTBot 3.6%
- .it (Italy) GPTBot 26.9%
- .ai GPTBot 6.6%
- .pl (Poland) GPTBot 26.9%
- .au (Australia) GPTBot 28.6%
- .tv GPTBot 10.2%
- .nl (Netherlands) GPTBot 27.1%
- .es (Spain) GPTBot 22.9%
- .co GPTBot 15.9%
- .ca (Canada) GPTBot 7.1%
- .ch (Switzerland) GPTBot 22.9%
- .ir (Iran) GPTBot 0%