AI Crawler Index · domain suffix
Do .ai websites block AI crawlers?
As of 2026-11-16, 6.6% of the 61 .ai websites whose robots.txt we could read block GPTBot, 4.9% block ClaudeBot and 1.6% block PerplexityBot (all 8,313 sites: 15.4%, 14.7%, 8.2%). 30 of them (49.2%) serve an llms.txt (all sites: 16.7%).
6.6%
block GPTBot (all sites 15.4%)
4.9%
block ClaudeBot (all sites 14.7%)
49.2%
serve an llms.txt (all sites 16.7%)
61
of 64 sites with a readable robots.txt
Block rate by crawler: .ai vs all sites
| Crawler | Operator | .ai | All sites | Difference |
|---|---|---|---|---|
| GPTBot | OpenAI | 6.6% | 15.4% | -8.8 pp |
| Amazonbot | Amazon | 4.9% | 13.2% | -8.3 pp |
| Applebot-Extended | Apple | 4.9% | 13.4% | -8.5 pp |
| Bytespider | ByteDance | 4.9% | 16.2% | -11.3 pp |
| ClaudeBot | Anthropic | 4.9% | 14.7% | -9.8 pp |
| Google-Extended | 4.9% | 13.2% | -8.3 pp | |
| CCBot | Common Crawl | 3.3% | 16.4% | -13.1 pp |
| ChatGPT-User | OpenAI | 3.3% | 7.4% | -4.1 pp |
| cohere-ai | Cohere | 3.3% | 12.7% | -9.4 pp |
| Google-CloudVertexBot | 3.3% | 8.8% | -5.5 pp | |
| Meta-ExternalAgent | Meta | 3.3% | 13.6% | -10.3 pp |
| MistralAI-Training | Mistral | 3.3% | 7.1% | -3.8 pp |
| OAI-SearchBot | OpenAI | 3.3% | 5.8% | -2.5 pp |
| Amzn-SearchBot | Amazon | 1.6% | 4.4% | -2.8 pp |
| Amzn-User | Amazon | 1.6% | 4.4% | -2.8 pp |
| Applebot | Apple | 1.6% | 3.2% | -1.6 pp |
| Claude-SearchBot | Anthropic | 1.6% | 5.7% | -4.1 pp |
| Claude-User | Anthropic | 1.6% | 5.9% | -4.3 pp |
| DuckAssistBot | DuckDuckGo | 1.6% | 5.9% | -4.3 pp |
| Google-Agent | 1.6% | 3.6% | -2 pp | |
| Google-GeminiNotebook | 1.6% | 3.3% | -1.7 pp | |
| meta-externalads | Meta | 1.6% | 2.9% | -1.3 pp |
| meta-externalfetcher | Meta | 1.6% | 6.5% | -4.9 pp |
| meta-webindexer | Meta | 1.6% | 5.1% | -3.5 pp |
| MistralAI-Index | Mistral | 1.6% | 3.6% | -2 pp |
| MistralAI-User | Mistral | 1.6% | 5.7% | -4.1 pp |
| OAI-AdsBot | OpenAI | 1.6% | 3% | -1.4 pp |
| Perplexity-User | Perplexity | 1.6% | 6.1% | -4.5 pp |
| PerplexityBot | Perplexity | 1.6% | 8.2% | -6.6 pp |
Who blocks what, by name: GPTBot, ClaudeBot, Google-Extended, Bytespider, Applebot-Extended.
Policy mix
| Policy | Sites | Share |
|---|---|---|
| Open | 48 | 75% |
| No robots.txt | 8 | 12.5% |
| Unreadable | 3 | 4.7% |
| Blocks some AI search | 2 | 3.1% |
| Blocks training only | 2 | 3.1% |
| Blocks all AI search | 1 | 1.6% |
Most restrictive .ai sites
- mgln.ai 29 of 29 blocked
- loudecho.ai 11 of 29 blocked
- claude.ai 4 of 29 blocked
- tripo3d.ai 3 of 29 blocked
- scite.ai 2 of 29 blocked
Sites with an llms.txt (30)
All 64 .ai sites
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 185 | claude.ai | Blocks some AI search | blocked | blocked | blocked | allowed | allowed | allowed | blocked | allowed | – |
| 300 | perplexity.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1203 | axon.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1490 | swishapps.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1613 | e-volution.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1702 | hybrid.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1878 | sardine.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2027 | bale.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2151 | openrouter.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2179 | polyspeak.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2259 | learnings.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2367 | vdo.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2520 | higgsfield.ai | Open | partial | partial | partial | partial | allowed | partial | partial | allowed | yes |
| 2709 | x.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2754 | seaart.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3393 | crushon.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3549 | beacons.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3621 | nrich.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3865 | drift-pixel.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3975 | opencode.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4015 | anota.ai | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4083 | programmaticx.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4153 | spicychat.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4283 | granola.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4313 | mistral.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4546 | z.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4719 | nexverse.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4748 | tripo3d.ai | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4760 | qwen.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4876 | landingsite.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 5050 | loudecho.ai | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 5287 | otter.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 5979 | verisoul.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 6124 | turboscribe.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6143 | scite.ai | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | allowed | yes |
| 6178 | masteredu.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6258 | mgln.ai | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 6666 | krisp.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 6935 | ourdream.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 7158 | engenius.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7264 | securiti.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 7486 | linke.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7532 | bluexad.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7554 | candy.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7620 | gmgn.ai | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7679 | superops.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 7763 | jasper.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 7791 | shortpixel.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7969 | luciq.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 7991 | trae.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8150 | arena.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8384 | dootchi.ai | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8572 | openart.ai | Open | partial | partial | partial | partial | partial | partial | partial | allowed | yes |
| 8654 | openclaw.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 8772 | kling.ai | Open | allowed | partial | partial | allowed | partial | partial | partial | allowed | yes |
| 8895 | ctrltech.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8954 | leonardo.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 9117 | devin.ai | Open | partial | partial | partial | partial | partial | partial | partial | allowed | yes |
| 9121 | dimensions.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9523 | lmstudio.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 9563 | datahive.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9657 | paradox.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 9779 | eightfold.ai | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 9786 | yunray.ai | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
Categories on .ai
FAQ
- Do .ai websites block AI crawlers?
- As of 2026-11-16, 6.6% of the 61 .ai websites whose robots.txt we could read block GPTBot, 4.9% block ClaudeBot and 1.6% block PerplexityBot (all 8,313 sites: 15.4%, 14.7%, 8.2%). 30 of them (49.2%) serve an llms.txt (all sites: 16.7%).
- Which .ai websites block the most AI crawlers?
- mgln.ai (29 of 29), loudecho.ai (11 of 29), claude.ai (4 of 29), tripo3d.ai (3 of 29), scite.ai (2 of 29). The full per-site status is on each site page.
- How is this measured?
- We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 64 .ai sites are in the sample, 61 had a readable robots.txt (3 did not answer or returned an error page).
Other countries and suffixes
- .com GPTBot 15.9%
- .org GPTBot 14.8%
- .net GPTBot 17.8%
- .ru (Russia) GPTBot 4.9%
- .de (Germany) GPTBot 24.9%
- .jp (Japan) GPTBot 6.9%
- .io GPTBot 7%
- .edu GPTBot 4.4%
- .gov GPTBot 4.7%
- .br (Brazil) GPTBot 8.1%
- .uk (United Kingdom) GPTBot 22%
- .fr (France) GPTBot 34.1%
- .in (India) GPTBot 5.6%
- .cn (China) GPTBot 3.6%
- .it (Italy) GPTBot 26.9%
- .pl (Poland) GPTBot 26.9%
- .au (Australia) GPTBot 28.6%
- .tv GPTBot 10.2%
- .nl (Netherlands) GPTBot 27.1%
- .es (Spain) GPTBot 22.9%
- .co GPTBot 15.9%
- .ca (Canada) GPTBot 7.1%
- .ch (Switzerland) GPTBot 22.9%
- .ir (Iran) GPTBot 0%