AI Crawler Index
29 AI crawlers and how many websites block them
Share of 853 popular websites (readable robots.txt) that block each AI crawler on the home page, as of 2026-10-01.
| Crawler | Operator | Purpose | Blocked | Top 100 |
|---|---|---|---|---|
| CCBot | Common Crawl | Training | 19.5% | 26.5% |
| Bytespider | ByteDance | Training | 17.7% | 30.1% |
| GPTBot | OpenAI | Training | 17% | 21.7% |
| ClaudeBot | Anthropic | Training | 16.3% | 22.9% |
| Meta-ExternalAgent | Meta | Training | 14.4% | 24.1% |
| Applebot-Extended | Apple | Training | 14.3% | 21.7% |
| Google-Extended | Training | 14% | 18.1% | |
| cohere-ai | Cohere | Training | 13.7% | 19.3% |
| Amazonbot | Amazon | Training | 13.1% | 18.1% |
| PerplexityBot | Perplexity | AI search | 12.7% | 21.7% |
| ChatGPT-User | OpenAI | User-triggered fetch | 9.6% | 15.7% |
| Perplexity-User | Perplexity | User-triggered fetch | 8.9% | 16.9% |
| meta-externalfetcher | Meta | User-triggered fetch | 8.7% | 14.5% |
| MistralAI-User | Mistral | User-triggered fetch | 8.4% | 14.5% |
| Claude-User | Anthropic | User-triggered fetch | 8.3% | 14.5% |
| Claude-SearchBot | Anthropic | AI search | 8.1% | 12% |
| DuckAssistBot | DuckDuckGo | AI search | 8.1% | 15.7% |
| Google-CloudVertexBot | AI search | 7.2% | 16.9% | |
| OAI-SearchBot | OpenAI | AI search | 6.7% | 12% |
| meta-webindexer | Meta | AI search | 6.3% | 14.5% |
| Amzn-SearchBot | Amazon | AI search | 5.9% | 13.3% |
| Amzn-User | Amazon | User-triggered fetch | 5.5% | 12% |
| MistralAI-Index | Mistral | AI search | 4.7% | 10.8% |
| MistralAI-Training | Mistral | Training | 4.7% | 10.8% |
| Google-Agent | User-triggered fetch | 4.5% | 10.8% | |
| OAI-AdsBot | OpenAI | Ads review | 4.3% | 12% |
| Google-GeminiNotebook | User-triggered fetch | 4.3% | 10.8% | |
| meta-externalads | Meta | Ads review | 3.8% | 9.6% |
| Applebot | Apple | AI search | 2.6% | 1.2% |