AI Crawler Index
29 AI crawlers and how many websites block them
Share of 8,313 popular websites (readable robots.txt) that block each AI crawler on the home page, as of 2026-11-16.
| Crawler | Operator | Purpose | Blocked | Top 100 |
|---|---|---|---|---|
| CCBot | Common Crawl | Training | 16.4% | 27.7% |
| Bytespider | ByteDance | Training | 16.2% | 30.1% |
| GPTBot | OpenAI | Training | 15.4% | 22.9% |
| ClaudeBot | Anthropic | Training | 14.7% | 25.3% |
| Meta-ExternalAgent | Meta | Training | 13.6% | 25.3% |
| Applebot-Extended | Apple | Training | 13.4% | 25.3% |
| Google-Extended | Training | 13.2% | 20.5% | |
| Amazonbot | Amazon | Training | 13.2% | 20.5% |
| cohere-ai | Cohere | Training | 12.7% | 19.3% |
| Google-CloudVertexBot | AI search | 8.8% | 16.9% | |
| PerplexityBot | Perplexity | AI search | 8.2% | 22.9% |
| ChatGPT-User | OpenAI | User-triggered fetch | 7.4% | 15.7% |
| MistralAI-Training | Mistral | Training | 7.1% | 10.8% |
| meta-externalfetcher | Meta | User-triggered fetch | 6.5% | 15.7% |
| Perplexity-User | Perplexity | User-triggered fetch | 6.1% | 16.9% |
| Claude-User | Anthropic | User-triggered fetch | 5.9% | 14.5% |
| DuckAssistBot | DuckDuckGo | AI search | 5.9% | 16.9% |
| OAI-SearchBot | OpenAI | AI search | 5.8% | 12% |
| Claude-SearchBot | Anthropic | AI search | 5.7% | 13.3% |
| MistralAI-User | Mistral | User-triggered fetch | 5.7% | 14.5% |
| meta-webindexer | Meta | AI search | 5.1% | 14.5% |
| Amzn-SearchBot | Amazon | AI search | 4.4% | 13.3% |
| Amzn-User | Amazon | User-triggered fetch | 4.4% | 12% |
| Google-Agent | User-triggered fetch | 3.6% | 10.8% | |
| MistralAI-Index | Mistral | AI search | 3.6% | 10.8% |
| Google-GeminiNotebook | User-triggered fetch | 3.3% | 10.8% | |
| Applebot | Apple | AI search | 3.2% | 1.2% |
| OAI-AdsBot | OpenAI | Ads review | 3% | 12% |
| meta-externalads | Meta | Ads review | 2.9% | 9.6% |