AI Crawler Index · category
Do social websites block AI crawlers?
As of 2026-11-16, 37.3% of the 51 social websites whose robots.txt we could read block GPTBot, 33.3% block ClaudeBot and 27.5% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 56 publish an llms.txt file.
Block rate by crawler: social vs all sites
| Crawler | Operator | Social | All sites |
|---|---|---|---|
| GPTBot | OpenAI | 37.3% | 15.4% |
| ClaudeBot | Anthropic | 33.3% | 14.7% |
| Google-Extended | 33.3% | 13.2% | |
| Applebot-Extended | Apple | 33.3% | 13.4% |
| Meta-ExternalAgent | Meta | 33.3% | 13.6% |
| CCBot | Common Crawl | 33.3% | 16.4% |
| Amazonbot | Amazon | 29.4% | 13.2% |
| Bytespider | ByteDance | 29.4% | 16.2% |
| PerplexityBot | Perplexity | 27.5% | 8.2% |
| cohere-ai | Cohere | 25.5% | 12.7% |
| ChatGPT-User | OpenAI | 23.5% | 7.4% |
| Google-CloudVertexBot | 23.5% | 8.8% | |
| Claude-User | Anthropic | 21.6% | 5.9% |
| Perplexity-User | Perplexity | 21.6% | 6.1% |
| meta-externalfetcher | Meta | 21.6% | 6.5% |
| MistralAI-User | Mistral | 21.6% | 5.7% |
| MistralAI-Training | Mistral | 21.6% | 7.1% |
| OAI-SearchBot | OpenAI | 19.6% | 5.8% |
| OAI-AdsBot | OpenAI | 19.6% | 3% |
| Claude-SearchBot | Anthropic | 19.6% | 5.7% |
| Google-GeminiNotebook | 19.6% | 3.3% | |
| Google-Agent | 19.6% | 3.6% | |
| meta-webindexer | Meta | 19.6% | 5.1% |
| Amzn-SearchBot | Amazon | 19.6% | 4.4% |
| Amzn-User | Amazon | 19.6% | 4.4% |
| DuckAssistBot | DuckDuckGo | 19.6% | 5.9% |
| MistralAI-Index | Mistral | 19.6% | 3.6% |
| meta-externalads | Meta | 17.6% | 2.9% |
| Applebot | Apple | 2% | 3.2% |
All 56 social sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 3 | facebook.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 7 | instagram.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 9 | twitter.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 11 | linkedin.com | Blocks some AI search | blocked | partial | blocked | blocked | partial | blocked | blocked | blocked | – |
| 21 | x.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 23 | skype.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 25 | pinterest.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 26 | tiktok.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 27 | whatsapp.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | blocked | allowed | – |
| 44 | t.me | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 48 | reddit.com | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 55 | snapchat.com | Blocks some AI search | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 59 | vk.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 61 | discord.gg | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 66 | tumblr.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 74 | vk.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 103 | telegram.org | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 129 | weibo.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 131 | ok.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 173 | bsky.app | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 180 | line.me | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 225 | behance.net | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 229 | gotinder.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 253 | deviantart.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 264 | threads.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 312 | viber.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 343 | meta.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 352 | wattpad.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 353 | patreon.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | blocked | allowed | allowed | – |
| 358 | ameblo.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 383 | livejournal.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 393 | pikabu.ru | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 398 | myspace.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 426 | note.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 434 | change.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 456 | xing.com | Open | partial | allowed | allowed | partial | partial | partial | allowed | allowed | yes |
| 486 | ameba.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 489 | archiveofourown.org | Blocks some AI search | blocked | allowed | blocked | allowed | allowed | allowed | allowed | blocked | – |
| 513 | gofundme.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 525 | kickstarter.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 555 | fmkorea.com | Open | partial | allowed | allowed | partial | allowed | partial | partial | partial | – |
| 557 | dcinside.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | blocked | blocked | – |
| 580 | douyin.com | Open | partial | partial | partial | allowed | allowed | partial | partial | allowed | – |
| 589 | livedoor.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 599 | fanfiction.net | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 601 | dribbble.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 617 | ruliweb.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 619 | arca.live | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 671 | syosetu.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 672 | ficbook.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 718 | ptt.cc | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 721 | ninisite.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 730 | dogdrip.net | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 821 | badoo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 994 | meetup.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 1431 | nextdoor.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
FAQ
- Do social websites block AI crawlers?
- As of 2026-11-16, 37.3% of the 51 social websites whose robots.txt we could read block GPTBot, 33.3% block ClaudeBot and 27.5% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 56 publish an llms.txt file.
- Which social websites block GPTBot?
- facebook.com, instagram.com, twitter.com, linkedin.com, x.com, pinterest.com, tiktok.com, whatsapp.com, reddit.com, snapchat.com, and more (full list on this page).
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%